Search server ES cluster performance optimization system and method and electronic equipment
By introducing data collection, processing and machine learning modules into the search server ES cluster, dynamically adjusting the ES cluster parameter configuration, the problem of insufficient data writing performance in the existing technology is solved, and efficient and scalable performance optimization effect is achieved .
Patent Information
- Application Number
- CN202411408650.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-02-14
AI Technical Summary
It is difficult for the prior art to effectively adjust the parameter configuration of the search server ES cluster to improve data writing performance, especially in high load and high concurrency scenarios, which may lead to insufficient performance, increased latency and system crashes.
Provides a search server ES cluster performance optimization system, including data collection module, data processing module, machine learning module and decision-making module. By collecting the running data of the ES cluster, performing data cleaning and feature engineering, training a random forest regression model, and dynamically adjusting the parameter configuration of the ES cluster to optimize performance.
It realizes real-time adjustment of ES cluster parameter configuration according to load changes, improves overall performance, enhances operating efficiency and manageability, reduces operating costs, improves scalability and flexibility, and ensures long-term stable and efficient operation of the system.
Smart Images
Figure CN119441267A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of software development technology, and in particular to a search server ES cluster performance optimization system, method and electronic device. Background Art
[0002] Elastic Search (ES) is an open source distributed search and analysis engine used to store, search and analyze large amounts of data in real time, with high performance and powerful full-text search capabilities.
[0003] Generally, multiple servers can be used as nodes to form an ES cluster. In this way, ES can process large amounts of data, achieve fault tolerance and high availability, and ensure data security and reliability.
[0004] The data writing performance of the ES cluster is very important for scenarios such as data synchronization writing, log analysis, big data processing, and high concurrency. In the data synchronization writing scenario, if the data writing performance of the ES cluster is insufficient, it will slow down the speed of multiple nodes synchronously writing data to the ES cluster; in the log analysis scenario, if the data writing performance of the ES cluster is insufficient, it may increase the delay of logs, affecting real-time monitoring and troubleshooting; in the big data processing scenario (such as data analysis, machine learning, etc.), if the writing performance is poor, it may cause data import to take too long, affecting the entire data processing process; in the high concurrency scenario, if the writing performance of the ES cluster is insufficient, it may cause request accumulation, increased response delays, and even system crashes.
[0005] Currently, there is an urgent need for a method to adjust the parameter configuration strategy of the ES cluster according to the requirements of the ES cluster data writing performance. Summary of the invention
[0006] The present invention provides a search server ES cluster performance optimization system, method and electronic device to dynamically adjust the parameter configuration strategy of the current ES cluster to respond to load changes in real time and improve overall performance. The specific technical solution is as follows:
[0007] In a first aspect, the present application provides a search server ES cluster performance optimization system, including:
[0008] Data collection module, data processing module, machine learning module, and decision making module;
[0009] The data collection module is used to collect the operation data of the ES cluster, wherein the operation data includes the performance data of the ES cluster and the specific indicator data of the ES cluster;
[0010] The data processing module is used to perform data cleaning on the operation data, and create new features based on the data features of the operation data, and add the new features to the original operation data set;
[0011] The machine learning module is used to train the prediction model based on the original running data set, and use cross-validation and grid search to determine the optimal parameters of the prediction model to obtain a target prediction model;
[0012] The decision-making module is used to input the current operating data of the ES cluster into the target prediction model, and adjust the parameter configuration of the ES cluster based on the performance prediction result output by the target prediction model to optimize the performance of the ES cluster.
[0013] In a possible implementation, the system further includes an execution module and a display module:
[0014] The execution module is used to execute the parameter configuration of the ES cluster formulated by the decision-making module, and monitor the performance of the ES cluster in real time. When it is determined that the performance of the ES cluster is lower than the set performance, the parameter configuration of the ES cluster is rolled back;
[0015] The display module is used to display the performance indicators of the ES cluster, the performance prediction results predicted by the target prediction model and the adjustment suggestions given for the performance prediction results to the system administrator, and determine instructions based on the operations received from the system administrator to confirm whether to adjust the parameter configuration of the ES cluster according to the adjustment suggestions.
[0016] In a possible implementation, the data collection module is specifically used for:
[0017] Configure Metricbeat to run on each node of the ES cluster and collect the performance data according to the set collection cycle, wherein the performance data includes: CPU usage, memory usage, disk IO operations and network usage data;
[0018] The monitoring application program interface is called to collect the specific indicator data, wherein the characteristic indicator data includes: query response time, shard size and node load.
[0019] In a possible implementation, the data processing module is specifically used to:
[0020] Removing invalid data and / or missing data from the operating data to obtain cleaned operating data;
[0021] Extracting data features of the cleaned operating data, and standardizing the data features to obtain standardized data features;
[0022] The standardized data features are subjected to feature scaling, new feature creation and feature encoding to obtain the newly added features, and the newly added features are added to the original running data set.
[0023] In a possible implementation, the machine learning module is specifically used to:
[0024] Dividing the original running data set into a training set and a test set, and resampling the original running data set to obtain multiple training sets;
[0025] Constructing a corresponding decision tree for each training set in the plurality of training sets, wherein the features of the nodes of each decision tree are selected according to a preset feature selection rule;
[0026] Calculate the average value of the prediction results output by each decision tree, and use the average value as the performance prediction result output by the prediction model;
[0027] The prediction model is tested using the test set, and based on the error of the performance prediction result output by the prediction model, the optimal parameters of the prediction model are determined in combination with the grid search algorithm to obtain the target prediction model.
[0028] In a possible implementation, the execution module is specifically used to:
[0029] Before executing the parameter configuration of the ES cluster formulated by the decision-making module, a safety and health check is performed to verify the safety of adjusting the parameter configuration of the ES cluster;
[0030] Adjusting the parameter configuration of the ES cluster based on the parameter configuration sample;
[0031] The performance of the ES cluster after the parameter configuration of the ES cluster is adjusted is monitored in real time, and when it is determined that the performance of the ES cluster is lower than the set performance, the parameter configuration of the ES cluster is rolled back.
[0032] The optimization system provided by the present invention has the following beneficial effects:
[0033] 1. In the data processing module, feature engineering involves standardizing numerical data, creating new features, and converting categorical variables into a format that can be processed by machine learning algorithms. It can improve the accuracy and generalization of model predictions. In particular, feature scaling is an important preprocessing step for many machine learning algorithms (distance-based algorithms such as k-nearest neighbors and support vector machines). Encoding categorical variables is because most machine learning algorithms operate on numerical data and cannot directly process text data.
[0034] 2. In the machine learning module, the core advantage of the random forest algorithm is that it improves model performance and reduces overfitting by integrating multiple simple models (decision trees). The randomness of each tree comes from the random selection of data points (self-service sampling) and the random subset selection of features. Due to the randomness and integration characteristics of the algorithm, random forests can usually generalize well to new data and improve the generalization ability of the system.
[0035] 3. Through the random forest regression algorithm, data related to ES cluster performance is collected and analyzed, and the parameter configuration strategy of the current ES cluster is dynamically adjusted to respond to load changes in real time and improve overall performance. This forward-looking strategy not only greatly enhances the operating efficiency and manageability of the ES cluster, but also reduces operating costs while improving the scalability and elasticity of the ES cluster, ensuring the long-term stability and efficient operation of the system.
[0036] In a second aspect, the present application provides a search server ES cluster performance optimization method, comprising:
[0037] Collecting the operation data of the ES cluster, wherein the operation data includes the performance data of the ES cluster and the specific indicator data of the ES cluster;
[0038] Performing data cleaning on the operation data, creating new features based on data features of the operation data, and adding the new features to the original operation data set;
[0039] The prediction model is trained based on the original running data set, and the optimal parameters of the prediction model are determined by cross-validation and grid search to obtain a target prediction model;
[0040] The current operation data of the ES cluster is input into the target prediction model, and based on the performance prediction result output by the target prediction model, the parameter configuration of the ES cluster is adjusted to optimize the performance of the ES cluster.
[0041] Through the above method, the parameter configuration strategy of the current ES cluster is dynamically adjusted to respond to load changes in real time and improve overall performance, which can enhance the operating efficiency and manageability of the ES cluster. At the same time, while reducing operating costs, it also improves the scalability and elasticity of the ES cluster and ensures the long-term stability and efficient operation of the system.
[0042] In a possible implementation, after adjusting the parameter configuration of the ES cluster, the method further includes:
[0043] Execute the adjusted parameter configuration of the ES cluster, and monitor the performance of the ES cluster in real time. When it is determined that the performance of the ES cluster is lower than the set performance, roll back the parameter configuration of the ES cluster;
[0044] The performance indicators of the ES cluster, the performance prediction results predicted by the target prediction model and the adjustment suggestions given for the performance prediction results are displayed to the system administrator, and based on the instructions received from the system administrator, it is determined whether to adjust the parameter configuration of the ES cluster according to the adjustment suggestions.
[0045] Through the above method, it can be automatically or manually determined whether to adjust the parameter configuration of the ES cluster according to the adjustment suggestion given by the prediction model.
[0046] In a third aspect, the present application provides an electronic device, including:
[0047] Memory, used to store computer programs;
[0048] The processor is used to implement the steps of the above-mentioned method for optimizing the performance of the search server ES cluster when executing the computer program stored in the memory.
[0049] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for optimizing the performance of a search server ES cluster are implemented.
[0050] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 A schematic diagram of the structure of a search server ES cluster performance optimization system provided for this application;
[0052] Figure 2 Schematic diagram of the operation of the search server ES cluster performance optimization system provided for this application;
[0053] Figure 3 One of the flowcharts of a search server ES cluster performance optimization method provided in this application;
[0054] Figure 4 The second flowchart of a search server ES cluster performance optimization method provided in this application;
[0055] Figure 5 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.
[0057] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0058] Search Server (ES) is an open source distributed search and analysis engine used to store, search and analyze large amounts of data in real time, with high performance and powerful full-text search capabilities.
[0059] Generally, multiple servers can be used as nodes to form an ES cluster. In this way, ES can process large amounts of data, achieve fault tolerance and high availability, and ensure data security and reliability.
[0060] The data writing performance of the ES cluster is very important for scenarios such as data synchronization writing, log analysis, big data processing, and high concurrency.
[0061] In the data synchronization writing scenario, for example, in the order system of an e-commerce platform, multiple order service nodes need to write order data to the ES cluster; if the data writing performance of the ES cluster is insufficient, it will slow down the speed at which multiple nodes synchronously write data to the ES cluster.
[0062] In log analysis scenarios, real-time writing and indexing of logs are very critical. If the data writing performance of the ES cluster is insufficient, it may increase log latency and affect real-time monitoring and troubleshooting.
[0063] In a big data processing scenario, for example, a telecommunications operator needs to write a large number of user call records into an ES cluster. If the write performance is poor, data import may take too long, affecting the entire data processing process.
[0064] In high-concurrency scenarios, for example, during a popular event on a social media platform, users post a large number of updates and comments, and this data needs to be written to the ES cluster; if the write performance of the ES cluster is insufficient, it may lead to request accumulation, increased response delays, and even system crashes.
[0065] Currently, there is an urgent need for a method to adjust the parameter configuration strategy of the ES cluster according to the requirements of the ES cluster data writing performance.
[0066] In view of this, in order to adjust the parameter configuration strategy of the current ES cluster according to load changes (data writing performance requirements) and enhance the operating efficiency of the ES cluster, the present application provides a search server ES cluster performance optimization system, the system including: a data collection module, a data processing module, a machine learning module and a decision-making module; a data collection module, used to collect the operating data of the ES cluster; a data processing module, used to clean the operating data, and based on the data features of the operating data, create new features, and add the new features to the original operating data set; a machine learning module, used to train the prediction model based on the original operating data set, and use cross-validation and grid search to determine the optimal parameters of the prediction model to obtain a target prediction model; a decision-making module, used to input the current operating data of the ES cluster into the target prediction model, and adjust the parameter configuration of the ES cluster based on the performance prediction results output by the target prediction model to optimize the performance of the ES cluster.
[0067] The optimization system provided in this application can dynamically adjust the parameter configuration strategy of the current ES cluster to respond to load changes in real time and improve the overall performance of the system; it can not only enhance the operating efficiency of the ES cluster, but also reduce operating costs while improving the scalability and elasticity of the ES cluster, ensuring the long-term stable and efficient operation of the system.
[0068] Reference Figure 1 As shown, it is a search server ES cluster performance optimization system provided by an embodiment of the present application, the system includes: a data collection module 101, a data processing module 102, a machine learning module 103 and a decision-making module 104;
[0069] A data collection module is used to collect the operation data of the ES cluster, wherein the operation data includes the performance data of the ES cluster and the specific indicator data of the ES cluster;
[0070] A data processing module is used to clean the operation data and create new features based on the data features of the operation data, and add the new features to the original operation data set;
[0071] The machine learning module is used to train the prediction model based on the original running data set, and use cross-validation and grid search to determine the optimal parameters of the prediction model to obtain the target prediction model;
[0072] The decision-making module is used to input the current operating data of the ES cluster into the target prediction model, and adjust the parameter configuration of the ES cluster based on the performance prediction results output by the target prediction model to optimize the performance of the ES cluster.
[0073] See also Figure 2 As shown, it is a schematic diagram of the operation of the search server ES cluster performance optimization system provided by the present application. In the embodiment of the present application, the data collection module can collect the operation data of the ES cluster (including collecting the performance data of the ES cluster and the specific indicator data of the ES cluster) in the following two ways:
[0074] Method 1: Deploy and configure Metricbeat on all search server (Elastic search) nodes to collect system and ES cluster performance data according to the set collection period, for example, 1ms, 1s, 10s, 1min, etc. The performance data of the ES cluster includes but is not limited to write latency, write throughput, CPU usage, memory usage, disk IO operations, and network usage data.
[0075] Method 2: Configure the monitoring function of Elastic search and call the monitoring application program interface (API) of Elastic search to collect specific indicator data of the ES cluster. The specific indicator data of the ES cluster includes but is not limited to indicators such as query response time, shard size, and node load.
[0076] In addition, after collecting the performance data of the ES cluster and the specific indicator data of the ES cluster, the present application can send the performance data of the ES cluster and the specific indicator data of the ES cluster to the central Elastic search monitoring cluster for subsequent data calls.
[0077] In the embodiment of the present application, the specific implementation scheme of the data processing module is as follows:
[0078] Extract data from the central Elastic search monitoring cluster. Specifically, you can use the Elasticsearch client library elastic search-py to extract data. Then preprocess the extracted data. The preprocessing includes cleaning the data, for example, filling missing values in the data or removing outliers in the data. This application does not impose any specific restrictions on this.
[0079] Furthermore, missing values can be filled in different ways for different types of data. For example, missing values of numerical features can be filled with the mean value, and missing values of categorical features can be filled with the mode. In addition, outliers can be defined using the standard deviation.
[0080] After the extracted data is cleaned to obtain the cleaned running data, feature engineering can be performed on the cleaned running data to create new features. Specifically, the data features of the running data can be standardized to obtain standardized data features; then, feature scaling can be performed on the standardized data features, new features based on existing features can be created and encoded to obtain new features, and the new features can be added to the original running data set.
[0081] Through the above, new features can be generated, the training set of the prediction model can be increased, and the generalization ability of the prediction model can be improved.
[0082] In the embodiment of the present application, the specific implementation scheme of the machine learning module is as follows:
[0083] Select features and target variables; divide the original running data set into a training set and a test set; create a random forest regressor instance, where the target variable represents the performance prediction result predicted by the prediction model. The performance prediction result can be the average response time of the ES cluster query. This application does not impose specific restrictions on the performance prediction result.
[0084] Furthermore, the working steps of the random forest regressor (model) are as follows:
[0085] Automatic sampling: Resample the original running data set to generate multiple different training sets, where each new training set is obtained by replacing sampling from the original data set. Therefore, the same sample can appear multiple times in one training set.
[0086] Constructing a decision tree: For each of the multiple training sets, the algorithm constructs a decision tree corresponding to each training set. When constructing each decision tree, the features of each decision tree node are selected according to the preset feature selection rules. This process is called random feature selection.
[0087] Aggregate results: The prediction of random forest is obtained by averaging the prediction results of all decision trees. Therefore, the average of the prediction results output by each decision tree can be calculated and used as the performance prediction result output by the prediction model;
[0088] Prediction model training: In actual operation, various model parameters of the prediction model (such as the number and depth of decision trees, etc.) can be optimized through cross-validation and grid search. Specifically, the prediction model is tested using the divided test set, and the optimal parameters of the prediction model are determined based on the error between the performance prediction results output by the prediction model and the actual performance requirements, combined with the grid search algorithm, to obtain the target prediction model, wherein the grid search algorithm includes the steps of setting grid parameters, creating a grid search instance, executing grid search, printing the optimal parameters, using the target prediction model for prediction, and calculating the mean square error of the target prediction model.
[0089] In the embodiment of the present application, the specific implementation scheme of the decision-making module is as follows:
[0090] Input the current operation data of the ES cluster into the target prediction model to predict the current and future required ES cluster write performance, and formulate the parameter configuration strategy of the ES cluster based on the prediction results and business rules (i.e., based on the performance prediction results output by the target prediction model, adjust the parameter configuration of the ES cluster to optimize the performance of the ES cluster). For example, when the average query response time output by the target prediction model exceeds the set threshold, the system recommends increasing the number of shards or adjusting the shard location, etc. In addition, a decision tree can be used to determine which node the shard should be transferred to based on factors such as CPU load and memory usage.
[0091] In the embodiment of the present application, the optimization system further includes an execution module 105 and a display module 106;
[0092] The execution module is used to execute the parameter configuration of the ES cluster made by the decision-making module and monitor the performance of the ES cluster in real time. When it is determined that the performance of the ES cluster is lower than the set performance, the parameter configuration of the ES cluster is rolled back;
[0093] The display module is used to display the performance indicators of the ES cluster, the performance prediction results predicted by the target prediction model, and the adjustment suggestions given for the performance prediction results to the system administrator, and determine the instructions based on the operations received from the system administrator to confirm whether to adjust the parameter configuration of the ES cluster according to the adjustment suggestions.
[0094] In the embodiment of the present application, the specific implementation scheme of the execution module is as follows:
[0095] Before executing the parameter configuration of the ES cluster made by the decision-making module, perform security and health checks to verify the safety of adjusting the parameter configuration of the ES cluster;
[0096] Adjust the parameter configuration of the ES cluster based on the parameter configuration sample. In the specific implementation, you can use the Elasticsearch API to perform the configuration parameter adjustment operations recommended by the system;
[0097] The performance of the ES cluster after the parameter configuration of the ES cluster is adjusted is monitored in real time. When it is determined that the performance of the ES cluster is lower than the set performance, the parameter configuration of the ES cluster is rolled back or changed.
[0098] In addition, it should be noted that the functions of the decision-making module and the execution module can be borrowed from each other, and the functions of the two can be adaptively adjusted.
[0099] Through the above method, the sharding configuration strategy of the ES cluster can be adjusted in real time according to the changes in load to optimize the performance of the ES cluster.
[0100] In the embodiment of the present application, the specific implementation scheme of the display module is as follows:
[0101] Use open source analysis and visualization platforms to build dashboards that display real-time performance indicators of the ES cluster, performance prediction results predicted by the target prediction model, and adjustment suggestions for the performance prediction results (including historical adjustment records and current adjustment records); develop web applications that allow administrators to view sharding adjustment suggestions proposed by the prediction model, and provide system administrators with a user interface for manually executing or undoing operations. In addition, the administrator's decisions and adjustment results can be recorded and used as feedback for subsequent prediction model training; based on the system administrator's operational determination instructions received, confirm whether to adjust the ES cluster parameter configuration according to the adjustment suggestions.
[0102] By collecting and analyzing data related to ES cluster performance, including write latency, write throughput, CPU usage, memory usage, disk I / O operations, network bandwidth usage, JVM memory pressure, garbage collection activity, current cluster configuration parameters, etc., the parameter configuration strategy of the current ES cluster is dynamically adjusted to respond to load changes in real time and improve overall performance.
[0103] In the embodiment of the present application, the optimization system also has a continuous optimization and monitoring function (module), which can track the performance of the ES cluster and iterate the prediction model. When tracking the performance of the ES cluster, the performance of the ES cluster and the adjustment effect of the system are continuously monitored, and the above monitoring data is collected as input for the subsequent continuous learning and improvement of the prediction model; when iterating the prediction model, the model is regularly retrained to incorporate new data and feedback to ensure the accuracy of the prediction.
[0104] The present invention is beneficial in that:
[0105] 1. In the data processing module, feature engineering involves standardizing numerical data, creating new features, and converting categorical variables into a format that can be processed by machine learning algorithms. It can improve the accuracy and generalization of model predictions. In particular, feature scaling is an important preprocessing step for many machine learning algorithms (distance-based algorithms such as k-nearest neighbors and support vector machines). Encoding categorical variables is because most machine learning algorithms operate on numerical data and cannot directly process text data.
[0106] 2. In the machine learning module, the core advantage of the random forest algorithm is that it improves model performance and reduces overfitting by integrating multiple simple models (decision trees). The randomness of each decision tree comes from the random selection of data points (self-service sampling) and the random subset selection of features. Due to the randomness and integration characteristics of the algorithm, the random forest can generalize well to new data and improve the generalization ability of the system.
[0107] 3. Through the random forest regression algorithm, data related to ES cluster performance is collected and analyzed, and the parameter configuration strategy of the current ES cluster is dynamically adjusted to respond to load changes in real time and improve overall performance. This forward-looking strategy not only greatly enhances the operating efficiency and manageability of the ES cluster, but also reduces operating costs while improving the scalability and elasticity of the ES cluster, ensuring the long-term stability and efficient operation of the system.
[0108] Based on the system provided in the above embodiment, the embodiment of the present application also provides a search server ES cluster performance optimization method, such as Figure 3 The figure shows a flow chart of a search server ES cluster performance optimization method provided by an embodiment of the present application, the method comprising:
[0109] S301, collecting operation data of the ES cluster, wherein the operation data includes performance data of the ES cluster and specific indicator data of the ES cluster;
[0110] S302, performing data cleaning on the operation data, creating new features based on the data features of the operation data, and adding the new features to the original operation data set;
[0111] S303, training the prediction model based on the original running data set, and using cross validation and grid search to determine the optimal parameters of the prediction model to obtain a target prediction model;
[0112] S304: Input the current operation data of the ES cluster into the target prediction model, and adjust the parameter configuration of the ES cluster based on the performance prediction result output by the target prediction model to optimize the performance of the ES cluster.
[0113] The method provided in this application can be applied to Figure 1 In the system shown, the embodiments of the method provided in the present application can refer to the above-mentioned embodiment of a search server ES cluster performance optimization system, and the repeated parts will not be repeated here.
[0114] In one possible implementation, reference Figure 4 As shown, after adjusting the parameter configuration of the ES cluster, the method further includes:
[0115] S401, executing the adjusted parameter configuration of the ES cluster, and monitoring the performance of the ES cluster in real time, and when it is determined that the performance of the ES cluster is lower than the set performance, rolling back the parameter configuration of the ES cluster;
[0116] S402, showing the performance indicators of the ES cluster, the performance prediction results predicted by the target prediction model, and the adjustment suggestions given for the performance prediction results to the system administrator, and confirming whether to adjust the parameter configuration of the ES cluster according to the adjustment suggestions based on the instructions received from the system administrator.
[0117] Similarly, the embodiment of the method after adjusting the parameter configuration of the ES cluster can also refer to the embodiment of the search server ES cluster performance optimization system mentioned above, and the repeated parts are not repeated here.
[0118] Through the method provided in this application, the parameter configuration strategy of the current ES cluster can be dynamically adjusted to respond to load changes in real time and improve overall performance, thereby enhancing the operating efficiency and manageability of the ES cluster. At the same time, while reducing operating costs, it also improves the scalability and elasticity of the ES cluster, ensuring the long-term stability and efficient operation of the system.
[0119] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the function of the optimization system of the aforementioned search server ES cluster performance, referring to Figure 5 , the electronic device comprises:
[0120] At least one processor 501, and a memory 502 connected to the at least one processor 501. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 In the example, the processor 501 and the memory 502 are connected via a bus 500. The bus 500 is Figure 5 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 500 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 501 can also be called a controller, and there is no limitation on the name.
[0121] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the optimization method of the search server ES cluster performance discussed above by executing the instructions stored in the memory 502. The processor 501 can implement Figure 1 The functions of each module in the system are shown.
[0122] Among them, the processor 501 is the control center of the system, and can use various interfaces and lines to connect various parts of the entire control device. By running or executing instructions stored in the memory 502 and calling data stored in the memory 502, the various functions of the system and processing data, the system can be monitored as a whole.
[0123] In one possible design, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.
[0124] Processor 501 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the optimization method for the search server ES cluster performance disclosed in the embodiments of the present application can be directly embodied as a hardware processor execution, or a combination of hardware and software modules in the processor.
[0125] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0126] By designing and programming the processor 501, the code corresponding to the method for optimizing the performance of the search server ES cluster introduced in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 3 The steps of the method for optimizing the performance of the search server ES cluster in the embodiment shown are as follows: How to design and program the processor 501 is a technology well known to those skilled in the art and will not be described in detail here.
[0127] Based on the same inventive concept, an embodiment of the present application further provides a storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer executes the method for optimizing the performance of the search server ES cluster discussed above.
[0128] In some possible implementations, various aspects of the method for optimizing the performance of a search server ES cluster provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the method for optimizing the performance of a search server ES cluster according to various exemplary implementations of the present application described above in this specification.
[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0133] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A search server ES cluster performance optimization system, characterized in that: The system includes: a data collection module, a data processing module, a machine learning module and a decision-making module; The data collection module is used to collect the operation data of the ES cluster, wherein the operation data includes the performance data of the ES cluster and the specific indicator data of the ES cluster; The data processing module is used to perform data cleaning on the operation data, and create new features based on the data features of the operation data, and add the new features to the original operation data set; The machine learning module is used to train the prediction model based on the original running data set, and use cross-validation and grid search to determine the optimal parameters of the prediction model to obtain a target prediction model; The decision-making module is used to input the current operating data of the ES cluster into the target prediction model, and adjust the parameter configuration of the ES cluster based on the performance prediction result output by the target prediction model to optimize the performance of the ES cluster.
2. The system according to claim 1, characterized in that The system also includes an execution module and a display module: The execution module is used to execute the parameter configuration of the ES cluster formulated by the decision-making module, and monitor the performance of the ES cluster in real time. When it is determined that the performance of the ES cluster is lower than the set performance, the parameter configuration of the ES cluster is rolled back; The display module is used to display the performance indicators of the ES cluster, the performance prediction results predicted by the target prediction model and the adjustment suggestions given for the performance prediction results to the system administrator, and determine instructions based on the operations received from the system administrator to confirm whether to adjust the parameter configuration of the ES cluster according to the adjustment suggestions.
3. The system according to claim 1, characterized in that The data collection module is specifically used for: Configure Metricbeat to run on each node of the ES cluster and collect the performance data according to the set collection cycle, wherein the performance data includes: CPU usage, memory usage, disk IO operations and network usage data; The monitoring application program interface is called to collect the specific indicator data, wherein the characteristic indicator data includes: query response time, shard size and node load.
4. The system according to claim 1, characterized in that The data processing module is specifically used for: Removing invalid data and / or missing data from the operating data to obtain cleaned operating data; Extracting data features of the cleaned operating data, and standardizing the data features to obtain standardized data features; The standardized data features are subjected to feature scaling, new feature creation and feature encoding to obtain the newly added features, and the newly added features are added to the original running data set.
5. The system according to claim 1, wherein: The machine learning module is specifically used for: Dividing the original running data set into a training set and a test set, and resampling the original running data set to obtain multiple training sets; Constructing a corresponding decision tree for each training set in the plurality of training sets, wherein the features of the nodes of each decision tree are selected according to a preset feature selection rule; Calculate the average value of the prediction results output by each decision tree, and use the average value as the performance prediction result output by the prediction model; The prediction model is tested using the test set, and based on the error of the performance prediction result output by the prediction model, the optimal parameters of the prediction model are determined in combination with the grid search algorithm to obtain the target prediction model.
6. The system according to claim 2, characterized in that The execution module is specifically used for: Before executing the parameter configuration of the ES cluster formulated by the decision-making module, a safety and health check is performed to verify the safety of adjusting the parameter configuration of the ES cluster; Adjusting the parameter configuration of the ES cluster based on the parameter configuration sample; The performance of the ES cluster after the parameter configuration of the ES cluster is adjusted is monitored in real time, and when it is determined that the performance of the ES cluster is lower than the set performance, the parameter configuration of the ES cluster is rolled back.
7. A search server ES cluster performance optimization method, characterized in that: The method comprises: Collecting the operating data of the ES cluster, wherein the operating data includes the performance data of the ES cluster and the specific indicator data of the ES cluster; Performing data cleaning on the operation data, creating new features based on data features of the operation data, and adding the new features to the original operation data set; The prediction model is trained based on the original running data set, and the optimal parameters of the prediction model are determined by cross-validation and grid search to obtain a target prediction model; The current operation data of the ES cluster is input into the target prediction model, and based on the performance prediction result output by the target prediction model, the parameter configuration of the ES cluster is adjusted to optimize the performance of the ES cluster.
8. The method according to claim 7, characterized in that After adjusting the parameter configuration of the ES cluster, the method further includes: Execute the adjusted parameter configuration of the ES cluster, and monitor the performance of the ES cluster in real time. When it is determined that the performance of the ES cluster is lower than the set performance, roll back the parameter configuration of the ES cluster; The performance indicators of the ES cluster, the performance prediction results predicted by the target prediction model and the adjustment suggestions given for the performance prediction results are displayed to the system administrator, and based on the instructions received from the system administrator, it is determined whether to adjust the parameter configuration of the ES cluster according to the adjustment suggestions.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 7 to 8 when executing a computer program stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 7 to 8 are implemented.