Search server cluster performance optimization method and device

Abstract

The invention discloses a search server cluster performance optimization method and device. Comprising the steps of obtaining a first performance parameter set of at least one search server cluster in a target system and a second performance parameter set of at least one slow query request in each search server cluster; for each cluster, determining a first tuning index according to the first performance parameter set and the cluster optimization coefficient, and determining a second tuning index corresponding to the slow query request according to a second performance parameter set of the slow query request in the cluster; and determining the tuning grade of each slow query request in the target system according to the first tuning index of each cluster and the second tuning index of each slow query request, and executing corresponding optimization operation on each slow query request according to the tuning grade. The technical problem that the performance of the service system is poor due to the fact that the search server cluster with the worst performance in the service system and the slow query request influencing the performance in the search server cluster cannot be rapidly checked in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of operation and maintenance technology, and in particular to a method and device for optimizing the performance of a search server cluster. Background Art

[0002] At present, in CRM (Customer Relationship Management) systems, Elasticsearch (i.e., search server) clusters are usually used to store various types of data, such as customer order messages, SAOP query messages, SAOP order messages, operation behavior analysis data, real-name photos, receipt structured data, short-term snapshots of three-household archive data, logs, etc. However, since these data involve many subsystems and submodules, in order to avoid mutual influence between various types of data, they are often stored in multiple sets of Elasticsearch clusters. However, since some of these scenarios have extremely high performance requirements for Elasticsearch clusters, once the Elasticsearch cluster has performance problems, it will affect the efficiency of front-line business processing, trigger customer complaints, and have a negative impact on the company. Traditional operation and maintenance tools can neither quickly locate the cluster with the worst performance among multiple clusters, nor can they quickly locate the slowest query that affects performance the most in a single cluster, resulting in a lack of performance troubleshooting.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] An embodiment of the present application provides a search server cluster performance optimization method and device to at least solve the technical problem that related technologies cannot quickly identify the search server cluster with the worst performance in the business system and the slow query requests that affect performance in the search server cluster, resulting in poor performance of the business system.

[0005] According to one aspect of an embodiment of the present application, a search server cluster performance optimization method is provided, including: obtaining at least one set of first performance parameter sets corresponding to at least one search server cluster in a target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster; for each search server cluster, determining a first tuning index of the search server cluster based on the first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and determining a second tuning index of each slow query request in the search server cluster based on the second performance parameter set corresponding to each slow query request in the search server cluster; determining a tuning level of each slow query request in at least one search server cluster in the target system based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, and performing corresponding optimization operations on each slow query request in order from high to low tuning levels.

[0006] Optionally, at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster are obtained, including: obtaining query logs of each search server cluster in the target system, wherein the query logs at least record query times of multiple query requests; for each search server cluster, screening out at least one slow query request from the query logs according to a query time threshold of each query request, and determining a query timeout duration of each slow query request and a total query timeout duration of at least one slow query request according to the query time of each slow query request and the corresponding query time threshold, and using at least one query timeout duration and a total query timeout duration corresponding to at least one slow query request processed by the search server cluster as the first performance parameter set corresponding to the search server cluster; determining multiple second performance parameter sets corresponding to each slow query request from the query logs, wherein the multiple types of performance parameters in the second performance parameter set include at least one of the following: query type, number of occurrences, and number of hit documents.

[0007] Optionally, the process of determining the cluster optimization coefficient includes: for each search server cluster, obtaining scenario information of the business scenario applied by the search server cluster, wherein the scenario information at least includes: the importance level of the business scenario in the target system and the importance level of the role played by the search server cluster in the business scenario; determining the cluster optimization coefficient of the search server cluster based on the scenario information, wherein the higher the importance level of the business scenario in the target system and / or the higher the importance level of the role played by the search server cluster in the business scenario, the higher the cluster optimization coefficient of the search server cluster.

[0008] Optionally, a first tuning index of the search server cluster is determined based on a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, including: calculating a first sum of multiple time durations within the first performance parameter set corresponding to the search server cluster, multiplying the first sum by a preset first weight value to obtain the first performance parameter of the search server cluster, and determining the first tuning index of the search server cluster using the first performance parameter and the cluster optimization coefficient of the search server cluster; or, calculating a second sum of square values ​​of multiple time durations within the first performance parameter corresponding to the search server cluster, multiplying the second sum by a preset first weight value to obtain the first performance parameter of the search server cluster, and determining the first tuning index of the search server cluster using the first performance parameter and the cluster optimization coefficient of the search server cluster.

[0009] Optionally, a second tuning index for each slow query request is determined based on a second performance parameter set corresponding to each slow query request in the search server cluster, including: for each slow query request in the search server cluster, calculating a third sum of each performance parameter in the second performance parameter set corresponding to the slow query request, and multiplying the third sum by a preset second weight value to obtain the second tuning index for the slow query request; or, calculating a fourth sum of the square values ​​of each performance parameter value in the second performance parameter set corresponding to the slow query request, and multiplying the fourth sum by the preset second weight value to obtain the second tuning index for the slow query request.

[0010] Optionally, the tuning level of each slow query request in at least one search server cluster in the target system is determined based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, including: sorting the first tuning indexes of each search server cluster in the target system in descending order to obtain a first sorting result; sorting the second tuning indexes of multiple slow query requests in each search server cluster in turn in descending order according to the first sorting result to obtain a second sorting result; determining the tuning level of each slow query request in at least one search server cluster in the target system based on the second sorting result, wherein, the higher the ranking position of the slow query request in the second sorting result, the higher the tuning level of the slow query request.

[0011] Optionally, before performing corresponding optimization operations on each slow query request in order of tuning level from high to low, the method also includes: determining the request type of each slow query request; matching the request type of each slow query request with a preset statement optimization table to obtain an optimization operation set for each slow query request, wherein the statement optimization table includes a correspondence between multiple types of slow query requests and optimization operation sets; wherein the optimization operation set includes: adjusting query requests and adjusting cluster parameters, and adjusting query requests includes at least one of the following: adjusting the number of aggregate query layers, adjusting query wildcards, adjusting batch submission volume, adjusting index field types, adjusting index shards, and adjusting the number of replicas, and adjusting cluster parameters includes at least one of the following: adjusting heap memory parameters, adjusting cache area parameters, adjusting refresh strategy parameters, adjusting merge strategy parameters, and adjusting thread pool parameters.

[0012] According to another aspect of an embodiment of the present application, a search server cluster performance optimization device is also provided, including: an acquisition module, used to acquire at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster; a determination module, used to determine, for each search server cluster, a first tuning index of the search server cluster based on the first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and to determine a second tuning index of each slow query request based on the second performance parameter set corresponding to each slow query request in the search server cluster; an optimization module, used to determine a tuning level of each slow query request in at least one search server cluster in the target system based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, and to perform corresponding optimization operations on each slow query request in order of the tuning level from high to low.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned search server cluster performance optimization method by running the computer program.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein when the computer program is executed by a processor, the above-mentioned search server cluster performance optimization method is implemented.

[0015] In an embodiment of the present application, the first tuning parameters of each search server cluster are determined by analyzing at least one set of first performance parameters corresponding to at least one search server cluster and the cluster optimization coefficient of each search server cluster; at the same time, the second tuning parameters of each slow query request are determined by analyzing at least one set of second performance parameters corresponding to at least one slow query request in each search server cluster; then, the tuning levels of multiple slow query requests in the target system are determined according to the first tuning parameters of each search server cluster and the second tuning parameters of each slow query request in a single search server cluster; finally, the corresponding optimization operations are performed on each slow query request in order from high to low in tuning level. It can be seen that the technical solution of the embodiment of the present application achieves the purpose of efficiently analyzing and optimizing the performance of the search server cluster, thereby solving the technical problem that the related technology cannot quickly check the search server cluster with the worst performance in the business system and the slow query requests that affect the performance in the search server cluster, resulting in poor performance of the business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a search server cluster performance optimization method according to an embodiment of the present application;

[0018] Figure 2 It is a flowchart of an optional search server cluster performance optimization method according to an embodiment of the present application;

[0019] Figure 3 is a schematic structural diagram of an optional search server cluster performance optimization device according to an embodiment of the present application;

[0020] Figure 4 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0024] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0025] A cluster is composed of more than one host node and provides storage and search services. A cluster composed of multiple nodes has redundancy, which can ensure the overall availability of the service when one or more nodes fail.

[0026] Elasticsearch, referred to as ES, is a distributed, highly scalable, and highly real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed, and explored, making full use of the horizontal scalability of ES to make data more valuable in a production environment. The implementation principle of ES is mainly divided into the following steps: First, the user submits the data to the ES database, and then the corresponding statement is segmented through the word segmentation controller, and its weight and word segmentation results are stored in the data together. When the user searches for data, the results are ranked and scored according to the weight, and finally the returned results are presented to the user. In addition, ES is distributed, which means that the index can be divided into shards, each shard can have 0 or more copies, each node hosts one or more shards, and acts as a coordinator to delegate operations to the correct shard.

[0027] Example 1

[0028] In the related art, traditional operation and maintenance tools are deficient in performance troubleshooting. First, they cannot quickly locate the search server cluster with the worst performance among multiple search server clusters. Second, they cannot quickly locate the slow query request that most affects performance in a single search server cluster, resulting in poor performance of the business system. Therefore, in order to solve the above problems, the embodiments of the present application provide relevant solutions, which are described in detail below.

[0029] According to an embodiment of the present application, a method embodiment of a search server cluster performance optimization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a search server cluster performance optimization method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the search server cluster performance optimization method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the search server cluster performance optimization method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0035] In the above operating environment, Figure 2 is a flow chart of an optional search server cluster performance optimization method according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S206, wherein:

[0036] Step S202: obtaining at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster.

[0037] In the technical solution provided in step S202, the target system may be any business system that uses multiple search server clusters, such as a CRM (Customer Relationship Management) system, etc. The search server cluster may be an Elasticsearch (ES) cluster, which may play different roles in different scenarios of the business system, such as log analysis and monitoring, full-text search, data analysis and visualization, data storage and retrieval, and separation of hot and cold data.

[0038] For multiple search server clusters for business systems, the performance of the search server clusters will greatly affect the performance of the business system. Therefore, in order to improve the performance of the business system, the performance of each search server cluster in the business system and the performance of multiple slow query requests in a single search server cluster can be analyzed to locate the cluster with the worst performance in multiple clusters and the slow query request that most affects cluster performance in a single cluster. To this end, at least one set of first performance parameter sets corresponding to at least one search server cluster and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster can be obtained in advance.

[0039] Step S204, for each search server cluster, determine the first tuning index of the search server cluster based on the first performance parameter set corresponding to the search server cluster and the preset cluster optimization coefficient, and determine the second tuning index of each slow query request based on the second performance parameter set corresponding to each slow query request in the search server cluster.

[0040] In the technical solution provided in step S204, for a single search server cluster, a first tuning index of the search server cluster can be determined based on a first performance parameter set corresponding to the search server cluster and a cluster optimization coefficient preset by the search server cluster, wherein the cluster optimization system of the search server cluster can be preset. Similarly, for each slow query request within a single search server cluster, a second tuning index of the slow query request can be determined based on a second performance parameter set corresponding to the slow query request.

[0041] Step S206, determining the tuning level of each slow query request in multiple search server clusters in the target system based on the first tuning index of multiple search server clusters and the second tuning index of multiple slow query requests in each search server cluster, and performing corresponding optimization operations on each slow query request in turn according to the tuning level.

[0042] In the technical solution provided in step S206, the tuning level of each slow query request in multiple search server clusters in the target system can be determined according to the first tuning index of each search server cluster and the order of the second tuning indexes of multiple slow query requests in each search server cluster, wherein the higher the tuning level, the higher the order in which the slow query request is optimized; finally, considering that the search server cluster with the worst performance is often the performance bottleneck of the entire system, and the slow query requests in the search server cluster usually occupy a large amount of resources, such as CPU, memory or I / O, to affect other operations of the search server cluster, therefore, when optimizing multiple slow query requests in the target system one by one, the corresponding optimization operations can be performed on each slow query request in order from high to low tuning levels, that is, the slow query requests that most affect the cluster performance in the search server cluster with the worst performance are optimized first according to the optimization level of each slow query request, so as to gradually improve the performance of the cluster and the system.

[0043] The above method of this embodiment is further introduced below.

[0044] As an optional implementation, in the technical solution provided in the above step S202, the method may include:

[0045] Step S2021, obtaining the query logs of each search server cluster in the target system.

[0046] Specifically, multiple search server clusters in the target system are controlled to process multiple query statements simultaneously to obtain query logs of each search server cluster, wherein the query logs include detailed information of multiple query statements, including but not limited to: timestamp, log level, node name, index name, query time, query time threshold, document ID, and query content, etc. It should be noted that the multiple query statements processed by the above-mentioned search server clusters are all the same standard test statements, so that fair performance comparisons can be made between different search server clusters to accurately identify which search server cluster performs better or worse in processing queries.

[0047] Step S2022: For each search server cluster, at least one slow query request is screened out from the query log according to the query time threshold of each query request, and the query timeout duration of each slow query request and the total query timeout duration of at least one slow query request are determined according to the query time of each slow query request and the corresponding query time threshold, and at least one query timeout duration and the total query timeout duration corresponding to at least one slow query request in the search server cluster are used as the first performance parameter set corresponding to the search server cluster; multiple second performance parameter sets corresponding to each slow query request are determined from the query log, wherein the multiple types of performance parameters in the second performance parameter set include at least one of the following: query type, number of occurrences, and number of hit documents.

[0048] The query time threshold of the above query request is pre-set. Generally, the query time threshold of the slow query request can be set by the statement: index.search.slowlog.threshold.quer. For example, index.search.slowlog.threshold.query.warn:10s, that is, all queries exceeding 10s are recorded as slow query logs at the warn level; index.search.slowlog.threshold.query.debug:2s, that is, all queries exceeding 2s are recorded as slow query logs at the debug level.

[0049] Specifically, if the query time threshold of each query request is tv, the query time when the search server cluster i executes the jth slow query request is t ij , then the total query timeout duration of the search server cluster i executing the jth slow query request can be expressed as: c ij =t ij -tv; If the number of slow query requests is p, the total query timeout duration of search server cluster i executing p slow query requests can be expressed as: Then, the p query timeout durations c corresponding to the p slow query requests processed by the search server cluster i are calculated as follows: ij And the total query timeout duration c of p query requests i Combined into the first performance parameter set C i ={c i1 ,c i2 ,…,c ip ,c i}, and the first performance parameter set C iEach duration element in is greater than 0. In addition, a second performance parameter set of each of the p slow query requests is obtained from the query log of the search server cluster i, wherein the performance parameters include: request type (such as query, fetch, index), query type, number of occurrences, number of hit documents, etc.

[0050] It should be noted that if no slow query request is screened out from the query logs in the search server cluster using the query time thresholds of each query request, it means that the performance of the search server cluster is good and does not need to be optimized.

[0051] As an optional implementation, in the technical solution provided in the above step S204, for each search server cluster, the process of determining the corresponding cluster optimization coefficient may include:

[0052] Step 1: Obtain scenario information of the business scenario applied by the search server cluster, wherein the scenario information at least includes: the importance level of the business scenario in the target system and the importance level of the role played by the search server cluster in the business scenario;

[0053] Step 2: Determine the cluster optimization coefficient of the search server cluster based on the scenario information, wherein the higher the importance level of the business scenario in the target system and / or the higher the importance level of the role played by the search server cluster in the business scenario, the higher the cluster optimization coefficient of the search server cluster.

[0054] In this embodiment, different search server clusters can be applied to different business scenarios of the target system, such as log analysis and monitoring, full-text search, data analysis and visualization, data storage and retrieval, etc., and the importance of the roles they play in these business scenarios is also different, so the importance of the search server cluster in the business scenario of the target system can be measured by the cluster optimization coefficient. In other words, the higher the importance level of the business scenario in which the search server cluster is applied in the target system (i.e., the search server cluster is used in the core business scenario), and / or the higher the importance of the role played by the search server cluster in the business scenario, the higher the cluster optimization coefficient of the corresponding search server cluster.

[0055] As an optional implementation, in the technical solution provided in the above step S204, the first tuning index of each search server cluster may be determined according to the following method, including:

[0056] Calculate the first sum of multiple time durations in the first performance parameter set corresponding to the search server cluster, multiply the first sum by a preset first weight value to obtain the first performance parameter of the search server cluster, and determine the first tuning index of the search server cluster by the first performance parameter and the cluster optimization coefficient of the search server cluster.

[0057] As another optional implementation, in the technical solution provided in the above step S204, the first tuning index of each search server cluster may also be determined according to the following method, including:

[0058] Calculate the second sum of the square values ​​of multiple time lengths within the first performance parameter corresponding to the search server cluster, multiply the second sum by a preset first weight value to obtain the first performance parameter of the search server cluster, and determine the first tuning index of the search server cluster by the first performance parameter and the cluster optimization coefficient of the search server cluster.

[0059] As an optional implementation, in the technical solution provided in the above step S204, the second tuning index of each slow query request in the search server cluster may be determined according to the following method, including:

[0060] A third sum of each performance parameter in the second performance parameter set corresponding to the slow query request is calculated, and the third sum is multiplied by a preset second weight value to obtain a second tuning index for the slow query request.

[0061] As another optional implementation, in the technical solution provided in the above step S204, the second tuning index of each slow query request in the search server cluster may also be determined according to the following method, including:

[0062] A fourth sum of square values ​​of each performance parameter value in the second performance parameter set corresponding to the slow query request is calculated, and the fourth sum is multiplied by a preset second weight value to obtain a second tuning index for the slow query request.

[0063] Among them, the settings of the above-mentioned first weight value and the second weight value can be set according to the actual application scenario, and this application does not make any specific restrictions on the settings of the first weight value and the second weight value.

[0064] As an optional implementation, in the technical solution provided in the above step S206, the tuning levels of multiple slow query requests in multiple search server clusters in the target system may be determined according to the following method, including:

[0065] Step S2061, sorting the first tuning indexes of the search server clusters in the target system in descending order to obtain a first sorting result;

[0066] Step S2062, sorting the second tuning indexes of multiple slow query requests in each search server cluster in descending order according to the first sorting result to obtain a second sorting result;

[0067] Step S2063: Determine the tuning level of each slow query request in at least one search server cluster in the target system according to the second sorting result.

[0068] The larger the second tuning index of the slow query request, the higher its ranking position in the second ranking result, and the higher the corresponding tuning level. Then, according to the tuning level of each slow query request, the slow query request with a high second tuning index in the search server cluster with a high first tuning index can be optimized.

[0069] Optionally, in the technical solution provided in the above step S206, before performing corresponding optimization operations on each slow query request in order of tuning level from high to low, the optimization operation corresponding to each slow query request may also be determined according to the following method, including:

[0070] Step 1: Determine the request type of each slow query request (such as query, fetch, index, etc.);

[0071] Step 2: Match the request type of each slow query request with the preset statement optimization table to obtain the optimization operation set of each slow query request, wherein the statement optimization table includes the correspondence between multiple types of slow query requests and the optimization operation sets.

[0072] In this embodiment, corresponding optimization operation sets are pre-set for different types of query requests, and different optimization operation sets include adjusting query requests and adjusting cluster parameters, wherein adjusting query requests includes at least one of the following: adjusting the number of aggregate query layers, adjusting query wildcards, adjusting batch submission volume, adjusting index field types, adjusting index shards, and adjusting the number of replicas, and adjusting cluster parameters includes at least one of the following: adjusting heap memory parameters, adjusting cache parameters, adjusting refresh strategy parameters, adjusting merge strategy parameters, and adjusting thread pool parameters. The settings of the optimization operation sets corresponding to specific different query requests can be set by yourself according to the actual application scenario.

[0073] Furthermore, after determining the optimization operation set corresponding to each slow query request, the optimization operation set corresponding to each slow query request can be fed back to the manager's client in sequence. The manager can choose to directly execute the optimization operation based on the fed back optimization operation set to optimize the corresponding slow query request; or, the manager can modify the fed back optimization operation set and execute the modified optimization operation set to optimize the corresponding slow query request.

[0074] Based on the scheme defined by the above steps S202 to S206, it can be known that in an embodiment, the first tuning parameters of each search server cluster are determined by analyzing at least one set of first performance parameters corresponding to at least one search server cluster and the cluster optimization coefficient of each search server cluster; at the same time, the second tuning parameters of each slow query request are determined by analyzing at least one set of second performance parameters corresponding to at least one slow query request in each search server cluster; then, the tuning level of multiple slow query requests in the target system is determined according to the first tuning parameters of each search server cluster and the second tuning parameters of each slow query request in a single search server cluster; finally, the corresponding optimization operations are performed on each slow query request in order from high to low in tuning level. It can be seen that the technical scheme of the embodiment of the present application achieves the purpose of efficiently analyzing and optimizing the performance of the search server cluster, thereby solving the technical problem that the related technology cannot quickly check the search server cluster with the worst performance in the business system and the slow query requests that affect the performance in the search server cluster, resulting in poor performance of the business system.

[0075] Example 2

[0076] Based on Embodiment 1 of the present application, an embodiment of a search server cluster performance optimization device is also provided, and the search server cluster performance optimization method of the above embodiment is executed when the device is running. Figure 3 is a schematic diagram of the structure of an optional search server cluster performance optimization device according to an embodiment of the present application, such as Figure 3 As shown, the search server cluster performance optimization device at least includes an acquisition module 32, a determination module 34 and an optimization module 36, wherein:

[0077] The acquisition module 32 is used to acquire at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster.

[0078] Optionally, the acquisition module 32 is also used to obtain query logs of each search server cluster in the target system; for each search server cluster, at least one slow query request is screened out from the query log based on the query time threshold of each query request, and the query timeout duration of each slow query request and the total query timeout duration of at least one slow query request are determined based on the query time of each slow query request and the corresponding query time threshold, and at least one query timeout duration and the total query timeout duration corresponding to at least one slow query request in the search server cluster are used as the first performance parameter set corresponding to the search server cluster; multiple second performance parameter sets corresponding to each slow query request are determined from the query log, wherein the multiple types of performance parameters in the second performance parameter set include at least one of the following: query type, number of occurrences, and number of hit documents.

[0079] Determination module 34 is used to determine, for each search server cluster, a first tuning index of the search server cluster based on a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and to determine a second tuning index of each slow query request based on a second performance parameter set corresponding to each slow query request in the search server cluster.

[0080] Specifically, the above-mentioned determination module 34 further includes: a first determination unit, a second determination unit and a third determination unit, wherein:

[0081] The first determining unit is used to determine the cluster optimization coefficient of the search server cluster by the following method, including:

[0082] Step 1: Obtain scenario information of the business scenario applied by the search server cluster, wherein the scenario information at least includes: the importance level of the business scenario in the target system and the importance level of the role played by the search server cluster in the business scenario;

[0083] Step 2: Determine the cluster optimization coefficient of the search server cluster based on the scenario information, wherein the higher the importance level of the business scenario in the target system and / or the higher the importance level of the role played by the search server cluster in the business scenario, the higher the cluster optimization coefficient of the search server cluster.

[0084] The second determining unit is used to determine the first tuning index of each search server cluster according to the following method, including:

[0085] Method 1: Calculate the first sum of multiple time durations in the first performance parameter set corresponding to the search server cluster, and multiply the first sum by a preset first weight value to obtain the first performance parameter of the search server cluster, and determine the first tuning index of the search server cluster by the first performance parameter and the cluster optimization coefficient of the search server cluster.

[0086] Method 2: Calculate the second sum of the square values ​​of multiple time lengths within the first performance parameter corresponding to the search server cluster, and multiply the second sum by a preset first weight value to obtain the first performance parameter of the search server cluster, and determine the first tuning index of the search server cluster by the first performance parameter and the cluster optimization coefficient of the search server cluster.

[0087] The third determining unit is used to determine the second tuning index of each slow query request in the search server cluster according to the following method, including:

[0088] Method 1: Calculate the third sum of each performance parameter in the second performance parameter set corresponding to the slow query request, and multiply the third sum by a preset second weight value to obtain the second tuning index of the slow query request.

[0089] Method 2: Calculate the fourth sum of the square values ​​of each performance parameter value in the second performance parameter set corresponding to the slow query request, and multiply the fourth sum by a preset second weight value to obtain the second tuning index of the slow query request.

[0090] The optimization module 36 is used to determine the tuning level of each slow query request in at least one search server cluster in the target system based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, and perform corresponding optimization operations on each slow query request in order from high to low tuning levels.

[0091] Optionally, the optimization module 36 includes: a fourth determination unit, a fifth determination unit, and an optimization operation feedback unit, wherein:

[0092] The fourth determination unit may determine the tuning levels of multiple slow query requests in multiple search server clusters in the target system according to the following method, including:

[0093] Step 1: sorting the first tuning indexes of each search server cluster in the target system in descending order to obtain a first sorting result;

[0094] Step 2: sort the second tuning indexes of multiple slow query requests in each search server cluster in descending order according to the first sorting result to obtain a second sorting result;

[0095] Step 3: Determine a tuning level for each slow query request in at least one search server cluster in the target system according to the second ranking result.

[0096] The fifth determining unit may determine the optimization operation corresponding to each slow query request according to the following method, including:

[0097] Step 1: Determine the request type of each slow query request (such as query, fetch, index, etc.);

[0098] Step 2: Match the request type of each slow query request with the preset statement optimization table to obtain the optimization operation set of each slow query request, wherein the statement optimization table includes the correspondence between multiple types of slow query requests and the optimization operation sets.

[0099] The above-mentioned optimization operation feedback unit can sequentially feed back the optimization operation sets corresponding to each slow query request determined by the fifth determination unit to the manager's client, and the manager can choose to directly execute the optimization operation according to the fed-back optimization operation set to optimize the corresponding slow query request; or, the manager can modify the fed-back optimization operation set and execute the modified optimization operation set to optimize the corresponding slow query request.

[0100] It should be noted that the various modules in the above-mentioned search server cluster performance optimization device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0101] Example 3

[0102] According to an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the search server cluster performance optimization method in Example 1.

[0103] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program:

[0104] Obtain at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster;

[0105] For each search server cluster, a first tuning index of the search server cluster is determined according to a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and a second tuning index of each slow query request is determined according to a second performance parameter set corresponding to each slow query request in the search server cluster;

[0106] Based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, the tuning level of each slow query request in at least one search server cluster in the target system is determined, and corresponding optimization operations are performed on each slow query request in order from high to low tuning levels.

[0107] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the search server cluster performance optimization method in Example 1 is implemented.

[0108] Optionally, the computer program executes the following steps:

[0109] Obtain at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster;

[0110] For each search server cluster, a first tuning index of the search server cluster is determined according to a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and a second tuning index of each slow query request is determined according to a second performance parameter set corresponding to each slow query request in the search server cluster;

[0111] Based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, the tuning level of each slow query request in at least one search server cluster in the target system is determined, and corresponding optimization operations are performed on each slow query request in order from high to low tuning levels.

[0112] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the search server cluster performance optimization method in Example 1 is executed when the program is running.

[0113] Optionally, the following steps are performed when the program is running:

[0114] Obtain at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster;

[0115] For each search server cluster, a first tuning index of the search server cluster is determined according to a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and a second tuning index of each slow query request is determined according to a second performance parameter set corresponding to each slow query request in the search server cluster;

[0116] Based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, the tuning level of each slow query request in at least one search server cluster in the target system is determined, and corresponding optimization operations are performed on each slow query request in order from high to low tuning levels.

[0117] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 4 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 4 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the search server cluster performance optimization method in the above-mentioned embodiment 1 when running.

[0118] Optionally, the processor is configured to implement the following steps by executing a computer program:

[0119] Obtain at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each search server cluster;

[0120] For each search server cluster, a first tuning index of the search server cluster is determined according to a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and a second tuning index of each slow query request is determined according to a second performance parameter set corresponding to each slow query request in the search server cluster;

[0121] Based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, the tuning level of each slow query request in at least one search server cluster in the target system is determined, and corresponding optimization operations are performed on each slow query request in order from high to low tuning levels.

[0122] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0123] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0128] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A search server cluster performance optimization method, characterized in that: include: Obtain at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each of the search server clusters; For each of the search server clusters, a first tuning index of the search server cluster is determined according to a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and a second tuning index of each of the slow query requests in the search server cluster is determined according to a second performance parameter set corresponding to each of the slow query requests in the search server cluster; The tuning level of each slow query request in at least one search server cluster in the target system is determined based on the first tuning index of each search server cluster and the second tuning index of at least one slow query request in each search server cluster, and corresponding optimization operations are performed on each slow query request in order of the tuning levels from high to low.

2. The method according to claim 1, characterized in that Obtaining at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each of the search server clusters, including: Obtaining query logs of each of the search server clusters in the target system, wherein the query logs at least record query times of multiple query requests; For each of the search server clusters, at least one slow query request is screened out from the query log according to the query time threshold of each of the query requests, and the query timeout duration of each of the slow query requests and the total query timeout duration of the at least one slow query request are determined according to the query time of each of the slow query requests and the corresponding query time threshold, and the at least one query timeout duration and the total query timeout duration corresponding to at least one of the slow query requests processed by the search server cluster are used as the first performance parameter set corresponding to the search server cluster; multiple second performance parameter sets corresponding to each of the slow query requests are determined from the query log, wherein the multiple types of performance parameters in the second performance parameter set include at least one of the following: query type, number of occurrences, and number of hit documents.

3. The method according to claim 1, characterized in that The process of determining the cluster optimization coefficient includes: For each of the search server clusters, scenario information of the business scenarios applied by the search server cluster is obtained, wherein the scenario information at least includes: the importance level of the business scenario in the target system and the importance level of the role played by the search server cluster in the business scenario; a cluster optimization coefficient of the search server cluster is determined based on the scenario information, wherein a higher importance level of the business scenario in the target system and / or a higher importance level of the role played by the search server cluster in the business scenario, a higher cluster optimization coefficient of the search server cluster.

4. The method according to claim 1, characterized in that: Determining a first tuning index of the search server cluster according to a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient includes: Calculate a first sum of multiple durations in a first performance parameter set corresponding to the search server cluster, and multiply the first sum by a preset first weight value to obtain a first performance parameter of the search server cluster, and determine a first tuning index of the search server cluster by combining the first performance parameter and a cluster optimization coefficient of the search server cluster; Alternatively, calculate the second sum of the square values ​​of multiple time lengths within the first performance parameter corresponding to the search server cluster, multiply the second sum by a preset first weight value to obtain the first performance parameter of the search server cluster, and determine the first tuning index of the search server cluster by combining the first performance parameter and the cluster optimization coefficient of the search server cluster.

5. The method according to claim 1, characterized in that Determining a second tuning index for each of the slow query requests according to a second performance parameter set corresponding to each of the slow query requests in the search server cluster includes: For each of the slow query requests in the search server cluster, calculate a third sum of each performance parameter in the second performance parameter set corresponding to the slow query request, and multiply the third sum by a preset second weight value to obtain a second tuning index for the slow query request; Alternatively, a fourth sum of square values ​​of each performance parameter value in a second performance parameter set corresponding to the slow query request is calculated, and the fourth sum is multiplied by a preset second weight value to obtain a second tuning index for the slow query request.

6. The method according to claim 1, characterized in that Determining a tuning level of each of the slow query requests in at least one of the search server clusters in the target system according to the first tuning index of each of the search server clusters and the second tuning index of at least one of the slow query requests in each of the search server clusters includes: Sorting the first tuning index of each search server cluster in the target system in descending order to obtain a first sorting result; According to the first sorting result, the second tuning indexes of the plurality of slow query requests in each search server cluster are sorted in descending order to obtain a second sorting result; The tuning level of each of the slow query requests in at least one of the search server clusters in the target system is determined based on the second sorting result, wherein the higher the ranking position of the slow query request in the second sorting result, the higher the tuning level of the slow query request.

7. The method according to claim 1, characterized in that Before performing corresponding optimization operations on each of the slow query requests in order of the tuning levels from high to low, the method further includes: Determine a request type of each slow query request; Matching the request type of each slow query request with a preset statement optimization table to obtain an optimization operation set for each slow query request, wherein the statement optimization table includes a correspondence between multiple types of slow query requests and optimization operation sets; Among them, the optimization operation set includes: adjusting query requests and adjusting cluster parameters, and the adjustment of query requests includes at least one of the following: adjusting the number of aggregate query layers, adjusting query wildcards, adjusting batch submission volume, adjusting index field types, adjusting index shards, and adjusting the number of replicas. The adjustment of cluster parameters includes at least one of the following: adjusting heap memory parameters, adjusting cache area parameters, adjusting refresh strategy parameters, adjusting merge strategy parameters, and adjusting thread pool parameters.

8. A search server cluster performance optimization device, characterized in that: include: An acquisition module, used to acquire at least one set of first performance parameter sets corresponding to at least one search server cluster in the target system and at least one set of second performance parameter sets corresponding to at least one slow query request in each of the search server clusters; A determination module, configured to determine, for each of the search server clusters, a first tuning index of the search server cluster according to a first performance parameter set corresponding to the search server cluster and a preset cluster optimization coefficient, and to determine a second tuning index of each of the slow query requests in the search server cluster according to a second performance parameter set corresponding to each of the slow query requests; An optimization module is used to determine the tuning level of each slow query request in at least one of the search server clusters in the target system based on the first tuning index of each of the search server clusters and the second tuning index of at least one of the slow query requests in each of the search server clusters, and perform corresponding optimization operations on each of the slow query requests in order of the tuning levels from high to low.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the search server cluster performance optimization method described in any one of claims 1 to 7 by running the computer program.

10. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the search server cluster performance optimization method described in any one of claims 1 to 7 is implemented.

Images