Method and device for configuring parameters of Elastic Search

Abstract

The invention provides an Elastic Search parameter configuration method and device, and the method comprises the steps: obtaining a key parameter set which affects the log read-write performance in a system, randomly endowing parameters in the key parameter set with parameter values, generating a plurality of parameter combinations as a primary population, and predicting the performance evaluation result of each parameter combination. And determining the fitness of the corresponding parameter combination based on the performance evaluation result of each parameter combination, performing selection, crossover and variation based on the fitness of each parameter combination, generating a new population, and iteratively executing the steps until a convergence condition is met, thereby obtaining the parameter combination with the highest fitness, according to the method, the parameter combination with the highest fitness is obtained, then the Elastic Search is configured on the basis of the parameter combination with the highest fitness, the automatic parameter configuration process is completed, performance modeling and an evolution search algorithm are fused, the method is suitable for various log-intensive system scenes, and the log read-write performance under various scenes is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of log access, and particularly to a method and device for parameter configuration of Elastic Search. Background Art

[0002] Elastic Search is a distributed search engine built on top of Lucene and is widely used in multiple business scenarios such as log analysis, full-text retrieval, and data monitoring. As a highly configurable system, Elasticsearch provides a rich set of system parameters for users to optimize, such as the number of shards, refresh interval, buffer size, thread pool configuration, cache policy, etc. These parameters have a significant impact on the query performance, indexing efficiency, memory occupancy, and resource utilization of the system. Therefore, in the AMHS digital solution for semiconductor factories, Elastic Search is used to achieve efficient storage and reading of logs.

[0003] However, in actual deployment, due to complex data structures, diverse business requirements, and dynamic changes in load types, parameter tuning has become a complex problem that is high-dimensional, non-linear, and context-dependent. Currently, most parameter configurations rely on operation and maintenance personnel to manually set based on experience, lacking a systematic tuning basis, and there is a problem of decreased log reading and writing performance due to unreasonable configurations. At the same time, traditional parameter search methods (such as grid search or brute-force testing) are costly and inefficient when faced with a high-dimensional parameter space and are difficult to adapt to the personalized needs of different scenarios. Therefore, there is an urgent need for an automated, generalized, data-driven parameter optimization method to automatically recommend the optimal parameter combination to optimize the reading and writing performance of Elastic Search in different log type scenarios. Summary of the Invention

[0004] The present invention provides a method and device for parameter configuration of Elastic Search to solve the defects of high cost and low efficiency in the existing parameter optimization solutions.

[0005] The present invention provides a method for parameter configuration of Elastic Search, including: Step 110, obtaining a set of key parameters that affect the log reading and writing performance in the Elastic Search system, and randomly assigning parameter values to the parameters in the set of key parameters to generate multiple parameter combinations as the initial population; Step 120, predicting the performance evaluation results of each parameter combination, and determining the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination; Step 130: Select based on the fitness of each parameter combination, perform gene exchange pairwise from the selected parameter combinations to generate the next generation of parameter combinations, and randomly change some gene positions in the next generation of parameter combinations with a preset probability; the gene position of any parameter combination corresponds to an Elastic Search parameter. Step 140: Generate a new population based on the next generation of parameter combinations, and iteratively execute Step 120, Step 130, and Step 140 until the convergence condition is reached to obtain the parameter combination with the highest fitness. Step 150: Configure Elastic Search based on the parameter combination with the highest fitness.

[0006] According to a method for configuring parameters of Elastic Search provided by the present invention, the performance evaluation result of any parameter combination is predicted based on the any parameter combination, data scenario structure characteristics, load characteristics, and environment characteristics. Among them, the data scenario structure characteristics include the number of fields for storing logs, field types, nesting depth, and data volume; the load characteristics include the proportions of read log requests and write log requests, and the request frequencies of read log requests and write log requests; the environment characteristics include the number of nodes of Elastic Search, the number of CPU cores, JVM configuration, and heap memory size.

[0007] According to a method for configuring parameters of Elastic Search provided by the present invention, predicting the performance evaluation result of each parameter combination includes: Input any parameter combination, data scenario structure characteristics, load characteristics, and environment characteristics into the trained performance evaluation model to obtain the performance evaluation result of the any parameter combination output by the trained performance evaluation model. Among them, in the model training stage, different parameter values are assigned to the parameters in the key parameter set to obtain multiple sample parameter combinations. Then, based on the standardized test tool, load tests are performed under different sample parameter combination configurations to obtain the performance metric values corresponding to each sample parameter combination. According to the difference between the performance evaluation result output by the performance evaluation model based on the sample parameter combination and the performance metric value corresponding to the sample parameter combination, the model parameters of the performance evaluation model are adjusted.

[0008] According to a method for configuring parameters of Elastic Search provided by the present invention, predicting the performance evaluation result of each parameter combination includes: Input any parameter combination, data scenario structure characteristics, load characteristics, and environment characteristics into the trained lightweight evaluation model to obtain the performance evaluation result of the any parameter combination output by the trained lightweight evaluation model. Wherein, during the training process of the lightweight evaluation model, different parameter values ​​are assigned to the parameters in the key parameter set. After obtaining a plurality of sample parameter combinations, load testing is performed based on a standardized test tool under different sample parameter combination configurations to obtain performance indicator values ​​corresponding to each sample parameter combination; according to the difference between the performance evaluation result output by a preset performance evaluation model based on the sample parameter combination and the performance indicator value corresponding to the sample parameter combination, the model parameters of the performance evaluation model are adjusted; according to the difference between the performance evaluation result output by the lightweight evaluation model based on the sample parameter combination and the performance evaluation result output by the performance evaluation model based on the sample parameter combination, the model parameters of the lightweight evaluation model are adjusted; wherein the operation speed of the lightweight evaluation model is faster than that of the performance evaluation model.

[0009] According to a parameter configuration method of Elastic Search provided by the present invention, determining the fitness of the corresponding parameter combination based on the performance evaluation result of each parameter combination includes: Sort each parameter combination in descending order according to the performance evaluation results; For the parameter combination ranked first, predicting a review performance evaluation result of the parameter combination ranked first based on the performance evaluation model; If the difference between the review performance evaluation result of the first-ranked parameter combination and the performance evaluation result is higher than a preset threshold, then predicting the review performance evaluation results of several parameter combinations ranked first based on the performance evaluation model; Based on the performance evaluation result of each parameter combination or the review performance evaluation result, the fitness of the corresponding parameter combination is determined.

[0010] According to a parameter configuration method of Elastic Search provided by the present invention, the performance evaluation model is built based on LightGBM, and the lightweight evaluation model is built based on a multi-layer perceptron.

[0011] According to a parameter configuration method of Elastic Search provided by the present invention, the parameters in the key parameter set include the number of index shards, refresh interval, search thread pool size, index buffer size and heap memory size.

[0012] The present invention also provides a parameter configuration device for Elastic Search, comprising: The first generation population generation unit is used to obtain a set of key parameters that affect log reading and writing performance in the Elastic Search system, and randomly assign parameter values ​​to the parameters in the key parameter set to generate multiple parameter combinations as the first generation population; A fitness determination unit for predicting the performance evaluation results of each parameter combination and determining the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination; A child individual generation unit for selecting based on the fitness of each parameter combination, performing gene exchange in pairs from the selected parameter combinations to generate the next generation of parameter combinations, and randomly changing some gene positions in the next generation of parameter combinations with a preset probability; each gene position of a parameter combination corresponds to an Elastic Search parameter; An iteration unit for generating a new population based on the next generation of parameter combinations and iteratively executing the functions implemented by the fitness determination unit, the child individual generation unit, and the iteration unit until a convergence condition is reached to obtain the parameter combination with the highest fitness; A configuration unit for configuring Elastic Search based on the parameter combination with the highest fitness.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the parameter configuration method of ElasticSearch as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the parameter configuration method of Elastic Search as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the parameter configuration method of Elastic Search as described in any one of the above.

[0016] The parameter configuration method and device for Elastic Search provided by the present invention obtain a set of key parameters that affect the log reading and writing performance in the Elastic Search system, randomly assign parameter values to the parameters in the set of key parameters to generate multiple parameter combinations as the initial population, predict the performance evaluation results of each parameter combination, determine the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination, and thus perform selection based on the fitness of each parameter combination, perform gene exchange in pairs from the selected parameter combinations to generate the next-generation parameter combinations, randomly change some gene positions in the next-generation parameter combinations with a preset probability, then generate a new population based on the next-generation parameter combinations, and iteratively execute the above steps until the convergence condition is reached to obtain the parameter combination with the highest fitness, and further configure Elastic Search based on the parameter combination with the highest fitness, completing the automatic configuration process of Elastic Search parameters. By integrating performance modeling and evolutionary search algorithms, it can quickly locate the optimal performance solution in a large-scale parameter space, is applicable to various log-intensive system scenarios, and improves the log reading and writing performance in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a flowchart of the parameter configuration method for Elastic Search provided by the present invention; Figure 2 It is a flowchart of the fitness determination method provided by the present invention; Figure 3 It is a schematic structural diagram of the parameter configuration device for Elastic Search provided by the present invention; Figure 4 It is a schematic structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0020] Figure 1 is a schematic flowchart of the parameter configuration method for Elastic Search provided by the present invention. As Figure 1 shown, the method includes: Step 110: Obtain a set of key parameters that affect the log reading and writing performance in the Elastic Search system, and randomly assign parameter values to the parameters in the set of key parameters to generate multiple parameter combinations as the initial population; Step 120: Predict the performance evaluation results of each parameter combination, and determine the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination; Step 130: Select based on the fitness of each parameter combination, and perform gene exchange in pairs from the selected parameter combinations to generate the next generation of parameter combinations, and randomly change some gene positions in the next generation of parameter combinations with a preset probability; Each gene position of any parameter combination corresponds to an Elastic Search parameter; Step 140: Generate a new population based on the next generation of parameter combinations, and iteratively execute Step 120, Step 130, and Step 140 until the convergence condition is reached to obtain the parameter combination with the highest fitness; Step 150: Configure Elastic Search based on the parameter combination with the highest fitness.

[0021] Specifically, in the Elastic Search system, there are a wide variety of parameters that affect the log reading and writing performance, and there are complex non-linear coupling relationships among different parameters. Therefore, in this embodiment, it is first necessary to identify and filter out a set of key parameters that have an impact on the reading and writing performance. This set can be determined through expert knowledge, historical tuning records, performance sensitivity analysis, or based on previous experiments. For example, this set of key parameters may include, but is not limited to, the following types of parameters: (1) Index structure parameters: such as number_of_shards (number of index shards), number_of_replicas (number of replicas); (2) Refresh and cache policy parameters: such as refresh_interval (refresh interval), indices.memory.index_buffer_size (index buffer size); (3) Thread pool and concurrency configuration parameters: such as thread_pool.search.size (size of the search thread pool), thread_pool.write.queue_size (size of the write thread pool); (4) JVM and heap configuration parameters: such as heap_size (heap memory size), young_generation_ratio; (5) Query cache related parameters: such as indices.queries.cache.size (index buffer size), indices.query.bool.max_clause_count (limit on the maximum number of clauses in a boolean query), etc.

[0022] After determining the set of key parameters, to construct the initial search space of the algorithm, a legal parameter value can be randomly assigned to each parameter in the above set of key parameters, and multiple parameter combinations are generated to form the initial population. The selection of parameter values can be discretely sampled according to a predefined range, or a representative value range can be constructed based on historical configuration experience to randomly select parameter values within the value range. Each parameter combination can be logically encoded as an "individual", and the internal parameter values are arranged in a certain order to form a "chromosome" structure, corresponding to the "gene locus" in an evolutionary algorithm (such as a genetic algorithm). For example, if 5 parameters are selected as optimization variables, each individual chromosome consists of 5 gene loci, respectively representing the values of the 5 parameters. The initial population can be composed of several (such as 100 or 200) individuals to ensure the initial coverage of the parameter space.

[0023] After the initial population is constructed, it is necessary to evaluate the performance of each parameter combination to measure its actual performance in ElasticSearch. However, it is too costly to directly deploy all parameter combinations one by one in the system and perform performance tests. Therefore, this embodiment introduces a performance prediction model based on machine learning to quickly predict the performance evaluation results of each parameter combination, and determines the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination, as the basis for its retention and reproduction in the evolutionary process. Among them, the performance evaluation result of any parameter combination includes one or more performance evaluation indicators. If the performance evaluation result of any parameter combination includes one performance evaluation indicator, the performance evaluation result can be directly used as the fitness. If the performance evaluation result of any parameter combination includes multiple performance evaluation indicators, such as query latency P95, write throughput, indexing rate, CPU usage, GC frequency, etc., it is necessary to calculate the fitness based on the business goals for these performance evaluation indicators. The fitness function can customize the weights of each performance evaluation indicator according to business requirements and calculate the fitness of the corresponding parameter combination in a weighted form: Fitness = w1×(1 / P95_latency) + w2×Throughput + w3×Indexing_speed-w4×CPU_usage -w5×GC_frequency Among them, w1 to w5 are preset weights, which are used to reflect the importance of different performance evaluation indicators in the target scenario. The higher the score, the stronger the fitness, indicating that the comprehensive performance of this parameter combination is better in the current scenario.

[0024] In the Elastic Search parameter optimization method proposed in the embodiment of the present invention, performance evaluation is the core link to achieve automatic parameter tuning. In order to improve the accuracy and efficiency of performance evaluation, this embodiment constructs a performance prediction model and uses machine learning technology to quickly estimate the system performance of any parameter combination in a specific business scenario, so as to reduce the tuning cost and test overhead. More importantly, it improves the generalization ability of performance evaluation to adapt to different data structures, load patterns and deployment environments. Therefore, in some embodiments, the performance evaluation result of any parameter combination is not based solely on the parameter value itself, but comprehensively considers the environmental characteristics such as the data scenario structure characteristics where the parameter combination is located, the current load characteristics of the system, and the resource configuration of the deployment environment on the basis of the parameter values of the parameter combination.

[0025] Specifically, the data scenario structure features describe the structural complexity of the log data indexed and stored in the current Elastic Search instance. These features include information such as the number of fields, field types, nesting depth, and total data volume. Among them, the number of fields refers to the number of key-value pairs contained in each log record, which directly affects the complexity of field mapping and matching during querying and indexing; the field type reflects the composition of the data types of each field, such as text, keyword, numeric, boolean, etc. Different field types correspond to different parsing methods and resource overheads, which have a significant impact on system performance; the nesting depth is used to measure the hierarchical complexity of the log structure, such as whether there are nested arrays, objects, multi-level indexes, etc. This nested structure will introduce additional index path calculation and search tree expansion in Elastic Search; and the data volume not only reflects the system storage pressure but also determines the basic resource consumption of querying and indexing operations.

[0026] In addition to the data structure, the real-time load borne during the system operation is also an important variable affecting the system performance under different parameter combinations. Therefore, this embodiment also introduces load features to reflect the composition and intensity of the read and write requests of the current system. The load features include the ratio of read requests to write requests, and their respective request frequencies (usually expressed in QPS). The requirements for read and write performance vary greatly at different time nodes, and this difference in read and write behavior will significantly affect the performance sensitivity of parameters such as thread pool configuration, cache policy, and refresh interval. Therefore, when predicting the performance evaluation results of parameter combinations, this read and write load feature is taken as a core consideration factor to help identify "whether the current scenario is more read-oriented or write-oriented", so as to more accurately judge the performance performance of a certain parameter combination under this load.

[0027] The environmental characteristics of the deployment environment are also indispensable measurement factors in the performance evaluation process. In the deployment of ElasticSearch, different resource conditions have different mechanisms of action on parameters. For example, a certain parameter combination may perform excellently on a cluster with rich resources, but may have mediocre effects or even cause performance degradation on resource-constrained edge nodes. Therefore, this embodiment also introduces environmental characteristics to describe the physical and logical resource status in the current Elastic Search running environment. The environmental characteristics include the number of Elastic Search nodes, the number of CPU cores, the JVM configuration, and the heap memory size. Among them, the number of Elastic Search nodes represents the scale of the system's distributed architecture and has a direct impact on shard allocation, parallel search, data replication, etc.; the number of CPU cores determines the number of threads that the system can process in parallel and is the basic resource affecting the efficiency of thread pools and compute-intensive operations; the JVM configuration includes GC algorithms, thread stack settings, class loading strategies, etc. Considering that ElasticSearch runs in a JAVA environment, the JVM configuration also affects the running performance of Elastic Search; the heap memory size directly determines the available upper limit of memory-related parameters such as cache capacity, index buffer, and write queue, and is also a key factor for system stability and performance.

[0028] Therefore, by fusing any parameter combination, data scenario structure characteristics, load characteristics, and environmental characteristics as the basis for performance evaluation, it can dynamically adapt to the actual business, data form, and system state, improving the performance evaluation accuracy of Elastic Search.

[0029] In some other embodiments, since a machine learning mechanism is adopted to construct a performance evaluation model to use the performance evaluation model to predict the performance evaluation results of any parameter combination, before actually using the performance evaluation model, the model needs to be trained. In the training stage, different parameter values can be assigned to the parameters in the key parameter set. After obtaining multiple sample parameter combinations, load tests are performed based on a standardized test tool under different sample parameter combination configurations to obtain the performance metric values corresponding to each sample parameter combination. Subsequently, according to the difference between the performance evaluation results output by the performance evaluation model based on the sample parameter combinations and the performance metric values corresponding to the sample parameter combinations, the model parameters of the performance evaluation model are adjusted. After the model training is completed, any parameter combination, data scenario structure characteristics, load characteristics, and environmental characteristics can be input into the trained performance evaluation model to obtain the performance evaluation results of this parameter combination output by the trained performance evaluation model.

[0030] Considering that the performance prediction model needs to be frequently used for the rapid evaluation of large-scale parameter combinations, especially when using genetic algorithms, the cost of each evaluation directly affects the search efficiency. Therefore, a lightweight proxy model (i.e., a lightweight evaluation model) can be constructed while training the main model (i.e., the performance evaluation model), and used to accelerate the evaluation in the search phase. Among them, the lightweight evaluation model operates faster than the performance evaluation model. In some embodiments, the performance evaluation model is built based on LightGBM, and the lightweight evaluation model is built based on a multi-layer perceptron. Specifically, in the model training phase, different parameter values ​​can be assigned to the parameters in the key parameter set to obtain multiple sample parameter combinations, and load tests can be performed under different sample parameter combination configurations based on standardized testing tools to obtain the performance index values ​​corresponding to each sample parameter combination, and then the performance evaluation results output by the preset performance evaluation model based on the sample parameter combination and the performance index values ​​corresponding to the corresponding sample parameter combination are adjusted according to the difference between the performance evaluation results and the performance index values ​​corresponding to the corresponding sample parameter combinations. The training of the performance evaluation model is completed. Subsequently, according to the difference between the performance evaluation results output by the lightweight evaluation model based on the sample parameter combination and the performance evaluation results output by the trained performance evaluation model based on the corresponding sample parameter combination, the model parameters of the lightweight evaluation model are adjusted to complete the training of the lightweight evaluation model. After the model training is completed, any parameter combination, data scenario structure characteristics, load characteristics, and environmental characteristics can be input into the trained lightweight evaluation model to obtain the performance evaluation result of the parameter combination output by the lightweight evaluation model.

[0031] On this basis, in order to take into account both the search efficiency and search accuracy of the optimal parameter combination, after obtaining the performance evaluation results of each parameter combination output by the lightweight evaluation model, such as Figure 2 As shown, the fitness of each parameter combination can be determined in the following way: Step 210, sorting each parameter combination in descending order according to the performance evaluation results; Step 220, for the parameter combination ranked first, predicting the review performance evaluation result of the parameter combination ranked first based on the performance evaluation model; Step 230, if the difference between the review performance evaluation result of the first-ranked parameter combination and the performance evaluation result is higher than a preset threshold, then predicting the review performance evaluation results of several parameter combinations ranked first based on the performance evaluation model; Step 240: Based on the performance evaluation result or the review performance evaluation result of each parameter combination, determine the fitness of the corresponding parameter combination.

[0032] Specifically, when sorting each parameter combination in descending order according to the performance evaluation results, if the performance evaluation results contain only one performance evaluation indicator, then each parameter combination is sorted directly according to the value of the performance evaluation indicator in the performance evaluation results; if the performance evaluation results contain multiple performance evaluation indicators, then the performance evaluation indicator with the highest degree of importance can be selected, and each parameter combination can be sorted according to the value of the performance evaluation indicator with the highest degree of importance. For the parameter combination ranked first, its performance evaluation result can be reviewed based on the performance evaluation model, and the model can be used to predict the reviewed performance evaluation result of the parameter combination ranked first again. If the difference between the reviewed performance evaluation result of the first-ranked parameter combination and the performance evaluation result output by the lightweight evaluation model is higher than a preset threshold (indicating that the lightweight evaluation model has a larger error than the performance evaluation model), the reviewed performance evaluation results of several parameter combinations ranked at the top are predicted again based on the performance evaluation model to correct the output result of the lightweight evaluation model, and then based on the performance evaluation result of each parameter combination (for the parameter combinations ranked later, they have not been reviewed, so there are only the performance evaluation results output by the lightweight evaluation model) or the reviewed performance evaluation results (for the parameter combinations ranked first, because they have been reviewed, the reviewed performance evaluation results output by the performance evaluation model are used), the fitness of the corresponding parameter combination is determined.

[0033] Based on the fitness of the parameter combination corresponding to each individual in the current population, the current population is subjected to evolutionary operations, including selection, crossover and mutation, to generate the next generation parameter combination to form the next generation population. Among them, the selection operation includes individual selection based on the fitness of the parameter combination corresponding to each individual, such as using the roulette method, tournament selection or elite retention strategy, and probabilistically selecting a certain proportion of individuals as "parents" according to the fitness, wherein the elite strategy can ensure that the current optimal individuals are retained to the next generation to avoid the loss of excellent solutions. The higher the fitness of the individual, the greater the probability of being selected, thereby accelerating the diffusion of high-quality individuals in the population. Subsequently, the crossover operation will perform gene exchange in pairs from the parameter combination corresponding to the selected individuals to generate the next generation parameter combination as the "offspring". For example, a single-point crossover or uniform crossover algorithm is used to perform gene exchange at one or several points on the chromosome to generate new individuals, in which the gene position of any parameter combination corresponds to an Elastic Search parameter. The crossover operation helps to integrate the excellent genes of excellent individuals and expand the search space. On this basis, in order to prevent falling into the local optimal solution, some gene positions in the new individual can be randomly changed with a small probability, where the mutation rate is low and can be maintained between 0.01 and 0.05.

[0034] After the new generation of individuals is generated, the next generation of population is formed. The new population can completely replace the old population, or a partial elite fusion strategy (such as retaining the old individuals with the top 10% fitness) can be adopted to improve stability. Subsequently, steps 120, 130, and 140 are re-executed for all individuals in the new population until the convergence condition is reached. Since this evolutionary process is iterated generation by generation, the following one or more convergence determination criteria can be set as the convergence condition: the best fitness value has not been improved for multiple consecutive generations; the standard deviation of the fitness of the entire population tends to converge; the preset number of iterations or time limit is reached; the best individual meets the performance threshold set by the business.

[0035] Once the convergence condition is met, it is considered that the search is completed, and the parameter combination with the highest fitness in the current population is output as the final optimization result, and Elastic Search is configured based on the parameter combination with the highest fitness. This configuration process can be completed automatically or provided as a recommended solution for the operation and maintenance personnel to verify the deployment.

[0036] In summary, the method provided by the embodiments of the present invention obtains a set of key parameters that affect the log reading and writing performance in the Elastic Search system, randomly assigns parameter values to the parameters in the set of key parameters to generate multiple parameter combinations as the initial population, predicts the performance evaluation results of each parameter combination, determines the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination, and thus selects based on the fitness of each parameter combination and performs gene exchange pairwise from the selected parameter combinations to generate the next generation of parameter combinations, and randomly changes some gene positions in the next generation of parameter combinations with a preset probability. Next, a new population is generated based on the next generation of parameter combinations, and the above steps are iteratively executed until the convergence condition is reached to obtain the parameter combination with the highest fitness. Furthermore, Elastic Search is configured based on the parameter combination with the highest fitness, completing the automatic configuration process of Elastic Search parameters. By integrating performance modeling and evolutionary search algorithms, it can quickly locate the optimal performance solution in a large-scale parameter space, is applicable to various log-intensive system scenarios, and improves the log reading and writing performance in various scenarios.

[0037] Next, the parameter configuration device of Elastic Search provided by the present invention will be described. The parameter configuration device of Elastic Search described below can be correspondingly referred to the parameter configuration method of Elastic Search described above.

[0038] Based on any of the above embodiments, Figure 3 is a schematic structural diagram of the parameter configuration device of Elastic Search provided by the present invention. As Figure 3 shown, the device includes: The initial population generation unit 310 is used to obtain a set of key parameters that affect the log reading and writing performance in the Elastic Search system, randomly assign parameter values to the parameters in the set of key parameters, and generate multiple parameter combinations as the initial population; The fitness determination unit 320 is used to predict the performance evaluation results of each parameter combination and determine the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination; The offspring individual generation unit 330 is used to make selections based on the fitness of each parameter combination, perform gene exchange in pairs from the selected parameter combinations to generate the next-generation parameter combinations, and randomly change some gene positions in the next-generation parameter combinations with a preset probability; each gene position of any parameter combination corresponds to an Elastic Search parameter; The iteration unit 340 is used to generate a new population based on the next-generation parameter combinations, and iteratively execute the functions implemented by the fitness determination unit, the offspring individual generation unit, and the iteration unit until the convergence condition is reached to obtain the parameter combination with the highest fitness; The configuration unit 350 is used to configure Elastic Search based on the parameter combination with the highest fitness.

[0039] The device provided by the embodiment of the present invention obtains a set of key parameters that affect the log reading and writing performance in the Elastic Search system, randomly assigns parameter values to the parameters in the set of key parameters, generates multiple parameter combinations as the initial population, predicts the performance evaluation results of each parameter combination, and determines the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination. Thus, selections are made based on the fitness of each parameter combination, gene exchange is performed in pairs from the selected parameter combinations to generate the next-generation parameter combinations, and some gene positions in the next-generation parameter combinations are randomly changed with a preset probability. Next, a new population is generated based on the next-generation parameter combinations, and the above steps are iteratively executed until the convergence condition is reached to obtain the parameter combination with the highest fitness. Furthermore, Elastic Search is configured based on the parameter combination with the highest fitness, completing the automatic configuration process of Elastic Search parameters. By integrating performance modeling and evolutionary search algorithms, it can quickly locate the optimal performance solution in a large-scale parameter space, is applicable to various log-intensive system scenarios, and improves the log reading and writing performance in various scenarios.

[0040] Based on any of the above embodiments, the performance evaluation result of any parameter combination is predicted based on the any parameter combination, data scenario structure characteristics, load characteristics, and environmental characteristics; Among them, the structural features of the data scenario include the number of fields for storing logs, field types, nesting depth, and data volume; the load features include the proportions of read log requests and write log requests, as well as the request frequencies of read log requests and write log requests; the environmental features include the number of nodes of Elastic Search, the number of CPU cores, JVM configuration, and heap memory size.

[0041] Based on any of the above embodiments, predicting the performance evaluation results of each parameter combination includes: Inputting any parameter combination, the structural features of the data scenario, the load features, and the environmental features into the trained performance evaluation model to obtain the performance evaluation results of the any parameter combination output by the trained performance evaluation model; Among them, in the model training stage, different parameter values are assigned to the parameters in the key parameter set to obtain multiple sample parameter combinations. Then, based on the standardized testing tool, load testing is performed under different sample parameter combination configurations to obtain the performance metric values corresponding to each sample parameter combination. According to the difference between the performance evaluation results output by the performance evaluation model based on the sample parameter combination and the performance metric values corresponding to the sample parameter combination, the model parameters of the performance evaluation model are adjusted.

[0042] Based on any of the above embodiments, predicting the performance evaluation results of each parameter combination includes: Inputting any parameter combination, the structural features of the data scenario, the load features, and the environmental features into the trained lightweight evaluation model to obtain the performance evaluation results of the any parameter combination output by the trained lightweight evaluation model; Among them, during the training process of the lightweight evaluation model, different parameter values are assigned to the parameters in the key parameter set to obtain multiple sample parameter combinations. Then, based on the standardized testing tool, load testing is performed under different sample parameter combination configurations to obtain the performance metric values corresponding to each sample parameter combination; according to the difference between the performance evaluation results output by the preset performance evaluation model based on the sample parameter combination and the performance metric values corresponding to the sample parameter combination, the model parameters of the performance evaluation model are adjusted; according to the difference between the performance evaluation results output by the lightweight evaluation model based on the sample parameter combination and the performance evaluation results output by the performance evaluation model based on the sample parameter combination, the model parameters of the lightweight evaluation model are adjusted; among them, the operation speed of the lightweight evaluation model is faster than that of the performance evaluation model.

[0043] Based on any of the above embodiments, determining the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination includes: Sorting each parameter combination in descending order according to the performance evaluation results; For the parameter combination ranked first, predicting a review performance evaluation result of the parameter combination ranked first based on the performance evaluation model; If the difference between the review performance evaluation result of the first-ranked parameter combination and the performance evaluation result is higher than a preset threshold, then predicting the review performance evaluation results of several parameter combinations ranked first based on the performance evaluation model; Based on the performance evaluation result of each parameter combination or the review performance evaluation result, the fitness of the corresponding parameter combination is determined.

[0044] Based on any of the above embodiments, the performance evaluation model is constructed based on LightGBM, and the lightweight evaluation model is constructed based on a multi-layer perceptron.

[0045] Based on any of the above embodiments, the parameters in the key parameter set include the number of index shards, refresh interval, search thread pool size, index buffer size and heap memory size.

[0046] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor (processor) 410, a memory (memory) 420, a communication interface (Communications Interface) 430 and a communication bus 440, wherein the processor 410, the memory 420, and the communication interface 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 420 to execute the parameter configuration method of Elastic Search, which includes: step 110, obtaining a set of key parameters that affect the log reading and writing performance in the Elastic Search system, and randomly assigning parameter values ​​to the parameters in the key parameter set to generate multiple parameter combinations as the initial population; step 120, predicting the performance evaluation result of each parameter combination, and determining the fitness of the corresponding parameter combination based on the performance evaluation result of each parameter combination; step 130, selecting based on the fitness of each parameter combination, and performing gene exchange in pairs from the selected parameter combinations to generate the next generation of parameter combinations, and randomly changing some gene bits in the next generation of parameter combinations with a preset probability; the gene bit of any parameter combination corresponds to an Elastic Search parameter; step 140, generating a new population based on the next generation of parameter combinations, and iteratively executing steps 120, 130 and 140 until the convergence condition is reached to obtain the parameter combination with the highest fitness; step 150, configuring Elastic Search based on the parameter combination with the highest fitness.

[0047] In addition, when the logical instructions in the above-mentioned memory 420 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0048] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the parameter configuration method of Elastic Search provided by the above-mentioned various methods. The method includes: Step 110, obtaining a set of key parameters that affect the log reading and writing performance in the Elastic Search system, and randomly assigning parameter values to the parameters in the set of key parameters to generate multiple parameter combinations as the initial population; Step 120, predicting the performance evaluation results of each parameter combination, and determining the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination; Step 130, performing selection based on the fitness of each parameter combination, and pairwise performing gene exchange on the selected parameter combinations to generate the next-generation parameter combinations, and randomly changing some gene positions in the next-generation parameter combinations with a preset probability; The gene position of any parameter combination corresponds to an Elastic Search parameter; Step 140, generating a new population based on the next-generation parameter combinations, and iteratively executing Step 120, Step 130, and Step 140 until the convergence condition is reached to obtain the parameter combination with the highest fitness; Step 150, configuring Elastic Search based on the parameter combination with the highest fitness.

[0049] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the parameter configuration method of Elastic Search provided above. The method includes: Step 110, obtaining a set of key parameters that affect the log reading and writing performance in the Elastic Search system, and randomly assigning parameter values to the parameters in the set of key parameters to generate multiple parameter combinations as the initial population; Step 120, predicting the performance evaluation results of each parameter combination, and determining the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination; Step 130, performing selection based on the fitness of each parameter combination, and pairwise exchanging genes from the selected parameter combinations to generate the next-generation parameter combinations, and randomly changing some gene positions in the next-generation parameter combinations with a preset probability; The gene position of any parameter combination corresponds to an Elastic Search parameter; Step 140, generating a new population based on the next-generation parameter combinations, and iteratively executing Step 120, Step 130, and Step 140 until the convergence condition is reached to obtain the parameter combination with the highest fitness; Step 150, configuring Elastic Search based on the parameter combination with the highest fitness.

[0050] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0051] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A parameter configuration method for Elastic Search, characterized in that, Including: Step 110: Obtain a set of key parameters that affect the log reading and writing performance in the Elastic Search system, and randomly assign parameter values to the parameters in the set of key parameters to generate multiple parameter combinations as the initial population; Step 120: Predict the performance evaluation results of each parameter combination, and determine the fitness of the corresponding parameter combination based on the performance evaluation results of each parameter combination; Step 130: Select based on the fitness of each parameter combination, perform gene exchange in pairs from the selected parameter combinations to generate the next generation of parameter combinations, and randomly change some gene positions in the next generation of parameter combinations with a preset probability; Each gene position of any parameter combination corresponds to an Elastic Search parameter; Step 140: Generate a new population based on the next generation of parameter combinations, and iteratively execute Step 120, Step 130, and Step 140 until the convergence condition is reached to obtain the parameter combination with the highest fitness; Step 150: Configure Elastic Search based on the parameter combination with the highest fitness.

2. The parameter configuration method of Elastic Search according to claim 1, wherein The performance evaluation result of any parameter combination is predicted based on the any parameter combination, data scenario structure characteristics, load characteristics, and environment characteristics; Among them, the data scenario structure characteristics include the number of fields for storing logs, field types, nesting depth, and data volume; the load characteristics include the proportion of read log requests and write log requests, and the request frequencies of read log requests and write log requests; the environment characteristics include the number of nodes of Elastic Search, the number of CPU cores, JVM configuration, and heap memory size.

3. The parameter configuration method of Elastic Search according to claim 2, characterized in that, The predicting the performance evaluation results of each parameter combination includes: Input any parameter combination, data scenario structure characteristics, load characteristics, and environment characteristics into the trained performance evaluation model to obtain the performance evaluation result of the any parameter combination output by the trained performance evaluation model; Among them, in the model training stage, different parameter values are assigned to the parameters in the set of key parameters to obtain multiple sample parameter combinations, and then load tests are performed based on the standardized test tool under different sample parameter combination configurations to obtain the performance metric values corresponding to each sample parameter combination, and according to the difference between the performance evaluation result output by the performance evaluation model based on the sample parameter combination and the performance metric value corresponding to the sample parameter combination, the model parameters of the performance evaluation model are adjusted.

4. The parameter configuration method of Elastic Search according to claim 2, characterized in that, The predicting the performance evaluation results of each parameter combination includes: Input any parameter combination, data scenario structure characteristics, load characteristics, and environment characteristics into the trained lightweight evaluation model to obtain the performance evaluation result of the any parameter combination output by the trained lightweight evaluation model; Wherein, during the training process of the lightweight evaluation model, different parameter values ​​are assigned to the parameters in the key parameter set. After obtaining a plurality of sample parameter combinations, load testing is performed based on a standardized test tool under different sample parameter combination configurations to obtain performance indicator values ​​corresponding to each sample parameter combination; according to the difference between the performance evaluation result output by a preset performance evaluation model based on the sample parameter combination and the performance indicator value corresponding to the sample parameter combination, the model parameters of the performance evaluation model are adjusted; according to the difference between the performance evaluation result output by the lightweight evaluation model based on the sample parameter combination and the performance evaluation result output by the performance evaluation model based on the sample parameter combination, the model parameters of the lightweight evaluation model are adjusted; wherein the operation speed of the lightweight evaluation model is faster than that of the performance evaluation model.

5. The parameter configuration method of Elastic Search according to claim 4, characterized in that, The step of determining the fitness of the corresponding parameter combination based on the performance evaluation result of each parameter combination includes: Sort each parameter combination in descending order according to the performance evaluation results; For the parameter combination ranked first, predicting a review performance evaluation result of the parameter combination ranked first based on the performance evaluation model; If the difference between the review performance evaluation result of the first-ranked parameter combination and the performance evaluation result is higher than a preset threshold, then predicting the review performance evaluation results of several parameter combinations ranked first based on the performance evaluation model; Based on the performance evaluation result of each parameter combination or the review performance evaluation result, the fitness of the corresponding parameter combination is determined.

6. The parameter configuration method of Elastic Search according to claim 4, wherein, The performance evaluation model is built based on LightGBM, and the lightweight evaluation model is built based on a multi-layer perceptron.

7. The parameter configuration method of Elastic Search according to any one of claims 1 to 6, characterized in that, The parameters in the key parameter set include the number of index shards, refresh interval, search thread pool size, index buffer size and heap memory size.

8. An Elastic Search parameter configuration device, characterized in that, include: The first generation population generation unit is used to obtain a set of key parameters that affect log reading and writing performance in the Elastic Search system, and randomly assign parameter values ​​to the parameters in the key parameter set to generate multiple parameter combinations as the first generation population; A fitness determination unit, used to predict the performance evaluation result of each parameter combination, and determine the fitness of the corresponding parameter combination based on the performance evaluation result of each parameter combination; The offspring individual generation unit is used to select based on the fitness of each parameter combination, and perform gene exchange in pairs from the selected parameter combinations to generate the next generation parameter combination, and randomly change some gene positions in the next generation parameter combination with a preset probability; the gene position of any parameter combination corresponds to an Elastic Search parameter; An iteration unit, used to generate a new population based on the next generation parameter combination, and iteratively execute the functions implemented by the fitness determination unit, the offspring individual generation unit and the iteration unit until a convergence condition is reached to obtain the parameter combination with the highest fitness; A configuration unit is used to configure Elastic Search based on the parameter combination with the highest fitness.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the program, it implements the parameter configuration method of Elastic Search as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the parameter configuration method of Elastic Search as described in any one of claims 1 to 7.

Images