Resource scheduling method, device and system for elastic search cluster

Abstract

The invention provides a resource scheduling method, device and system for an elastic search cluster, and belongs to the technical field of data storage. According to the scheme provided by the invention, the data writing rate of the target index can be counted, and when the data writing rate is out of the target rate range, the resource quantity of the nodes occupied by the multiple fragments of the target index can be dynamically adjusted. As the data storage capacity of the target index and the supported maximum data writing rate are both related to the resource quantity occupied by the multiple fragments, the scheme provided by the invention can be used for improving the resource utilization rate of the multiple fragments on the premise of not changing the number of the fragments included in the target index. And dynamic adjustment of the data storage capacity and the maximum data writing rate of the target index is realized. Therefore, not only is the flexibility of data writing effectively improved, but also the utilization rate of cluster resources is effectively improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202210764591.4, and the original application date is June 29, 2022. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of data storage technology, and in particular to a resource scheduling method, device and system for an elastic search cluster. Background Art

[0003] Elastic-search (ES) is a distributed, highly scalable and real-time data search engine that can provide data search services in clusters. A cluster usually includes multiple nodes, each of which can be a physical machine or a virtual machine (VM).

[0004] In an ES cluster, the smallest unit of data storage and search is a document, and the logical space (i.e., container) used to store documents is called an index. In order to increase the data storage capacity of the index and achieve high availability, the index can be divided into multiple shards, and at least one corresponding replica is created for each shard. Each shard and its corresponding at least one replica are distributed in different nodes. When storing a document in an index, it is necessary to first calculate the target shard for storing the document based on the document's identity (ID) and the number of shards included in the index. After that, the document can be stored in the target shard, and the target shard will synchronize the document to at least one corresponding replica. When it is necessary to read a document from the index, the target shard storing the document can be determined based on the document's ID and the number of shards included in the index, and read from the target shard.

[0005] Since the target shards need to be calculated based on the number of shards included in the index when storing documents in the index and reading documents from the index, the number of shards included in the index should be fixed. That is, after the index is created, the number of shards included in the index cannot be modified. As a result, the flexibility of data writing is low. Summary of the invention

[0006] The present application provides a resource scheduling method, device and system for an elastic search cluster, which can solve the technical problem of low data writing flexibility in an ES cluster.

[0007] In a first aspect, a resource scheduling method for an ES cluster is provided, the ES cluster comprising a plurality of first nodes, the plurality of first nodes being used to carry a plurality of shards of a target index. The method comprises: counting a data write rate of the target index, and adjusting the amount of resources of the nodes occupied by the plurality of shards of the target index based on the data write rate of the target index being outside a target rate range.

[0008] Since the data storage capacity of the target index and the maximum data writing rate supported are related to the amount of resources occupied by the multiple shards, the solution provided by the present application can dynamically adjust the data storage capacity and the maximum data writing rate of the target index without changing the number of shards included in the target index. As a result, not only the flexibility of data writing is effectively improved, but also the utilization rate of cluster resources is effectively improved.

[0009] Optionally, based on the data write rate of the target index being outside the target rate range, the process of adjusting the amount of resources of the nodes occupied by the multiple shards may include: based on the data write rate of the target index being greater than the upper limit of the target rate range, adding at least one second node to the ES cluster, and migrating at least one of the multiple shards to the at least one second node.

[0010] By adding at least one second node to the ES cluster, the ES cluster can be expanded and the amount of resources occupied by the target index can be increased. Thus, the data storage capacity and maximum data writing rate of the target index can be effectively improved, so that the data writing performance of the target index can match the current actual data writing rate.

[0011] Optionally, before adding at least one second node to the ES cluster, the method may further include: determining the number of at least one second node to be added based on the difference between the data write rate of the target index and the upper limit of the target rate range, wherein the number is positively correlated with the difference, and the node specifications of the at least one second node are all preset specifications.

[0012] The solution provided in the present application can add at least one second node of preset specifications in the ES cluster according to the difference in write rate. The method of adding the second node is relatively simple and efficient.

[0013] Optionally, before adding at least one second node to the ES cluster, the method may further include: determining the number of at least one second node to be added and the node specification of each second node based on the difference between the data write rate of the target index and the upper limit of the target rate range, and multiple different alternative node specifications. The node specification of each second node is selected from the specifications of the multiple alternative nodes.

[0014] If the available nodes in the resource pool have a variety of different node specifications, the solution provided by the present application can determine at least one second node from the available nodes of the multiple different node specifications based on the difference in write rates. The flexibility of the method of adding a second node is high. For example, the solution provided by the present application can determine at least one second node based on the cost of each alternative node specification using a cost-first strategy. Thus, the cost of the newly added node can be effectively reduced.

[0015] Optionally, the total number m1 of processor cores included in the at least one second node may satisfy: m1≥(s1-s2) / s0, where s1 is the data write rate of the target index, s2 is the upper limit of the target rate range, and s0 is the data write rate threshold of each processor core.

[0016] Based on the above formula, it can be ensured that after at least one second node is added, the data writing performance of multiple first nodes and at least one second node used to carry the target index can match the data writing rate of the target index.

[0017] Optionally, the resource used by the at least one second node may be a bidding resource. Since the cost of bidding resources is relatively low, the solution provided in this application may preferentially use bidding resources as resources of the newly added node.

[0018] Optionally, the at least one shard may satisfy at least one of the following conditions: the amount of data written in the shard is less than a first threshold; the total number of shards distributed in the first node where the shard is located is greater than a second threshold.

[0019] Based on the above conditions, the solution provided by the present application can select shards with a smaller amount of written data for migration, and / or select shards in the first node with a larger number of shards for migration. In this way, the migration efficiency of the shards can be effectively improved, the migration cost can be reduced, and the number of shards in each first node can be ensured to be relatively balanced.

[0020] Optionally, based on the data write rate of the target index being outside the target rate range, the process of adjusting the amount of resources of the nodes occupied by the multiple shards may include: based on the data write rate of the target index being less than the lower limit of the target rate range, determining at least one first target node to be deleted from the multiple first nodes; migrating all shards distributed in the at least one first target node to other first nodes except the at least one first target node, and deleting the at least one first target node.

[0021] If the data write rate of the target index is less than the lower limit of the target rate range, it indicates that the data write performance of multiple first nodes in the ES cluster is not fully utilized. Therefore, the capacity of the ES cluster can be reduced by deleting at least one first node, thereby effectively improving the resource utilization of the ES cluster and reducing the cost of the ES cluster.

[0022] Optionally, the total number of shards distributed in each first target node is less than a third threshold. That is, the solution provided by the present application can delete the first node with a small number of shards. Thus, it can be ensured that before deleting the at least one first target node, the shards in the at least one first target node can be quickly migrated to other first nodes.

[0023] Optionally, based on the data write rate of the target index being outside the target rate range, the process of adjusting the amount of resources of the nodes occupied by the multiple shards may include: based on the data write rate of the target index being outside the target rate range, adjusting the node specification of at least one first node among the multiple first nodes.

[0024] The solution provided in the present application can also adjust the amount of resources occupied by the target index by dynamically adjusting the node specifications of the first node without changing the first node to which the shard belongs.

[0025] Optionally, based on the data write rate of the target index being outside the target rate range, the process of adjusting the node specification of at least one of the multiple first nodes may include: based on the data write rate of the target index being greater than the upper limit of the target rate range, increasing the node specification of at least one second target node of the multiple first nodes. The at least one second target node may be a first node with a large number of shards in the ES cluster.

[0026] Optionally, based on the data write rate of the target index being outside the target rate range, the process of adjusting the node specification of at least one of the multiple first nodes may include: based on the data write rate of the target index being less than the lower limit of the target rate range, reducing the node specification of at least one third target node of the multiple first nodes. The at least one third target node may be a first node with a smaller number of shards in the ES cluster.

[0027] Optionally, the resource used by the fourth target node among the multiple first nodes is a bidding resource. The method may also include: adding at least one third node to the ES cluster based on the remaining available duration of the bidding resource used by the fourth target node being less than a duration threshold; migrating all shards distributed in the fourth target node to the at least one third node, and deleting the third target node.

[0028] Since the cost of bidding resources is relatively low, the solution provided by this application can use bidding resources as resources of the fourth target node, and can migrate the shards in the fourth target node to the newly added third node in a timely manner when the remaining usage time of the bidding resources is less than the time threshold. In this way, the cost of the ES cluster can be effectively reduced while ensuring the data reading and writing performance of the ES cluster.

[0029] Optionally, the resources used by the at least one third node may include on-demand resources and / or competitive resources. Among them, on-demand resources refer to resources purchased according to business needs, and their cost is higher than competitive resources. For example, the solution provided in this application can determine the type of resources used by the at least one third node according to a cost-first strategy.

[0030] Optionally, the ES cluster may further include multiple fourth nodes, which are used to carry multiple copies of the target index. The method may also include: counting the data reading frequency of the target index, and adjusting the number of copies included in the target index based on the data reading frequency of the target index being outside the target frequency range.

[0031] The solution provided in this application can adjust the number of copies included in the target index so that the number of copies included in the target index can match the data reading frequency. Thus, the resource utilization of the ES cluster can be effectively improved while ensuring data reading performance.

[0032] Optionally, based on the data reading frequency of the target index being outside the target frequency range, the process of adjusting the number of replicas included in the target index may include: based on the data reading frequency of the target index being greater than the upper limit of the target frequency range, adding at least one fifth node to the ES cluster, and adding at least one replica of the target index in each fifth node.

[0033] By adding a fifth node to the ES cluster and adding at least one replica to the fifth node, the data reading performance of the ES cluster can be effectively improved.

[0034] Optionally, based on the data reading frequency of the target index being outside the target frequency range, the process of adjusting the number of replicas included in the target index may include: based on the data reading frequency of the target index being less than the lower limit of the target frequency range, deleting the replicas distributed in at least one fifth target node among the multiple fourth nodes, and then deleting the at least one fifth target node.

[0035] If the data reading frequency of the target index is less than the lower limit of the target frequency range, it indicates that the data reading performance of multiple fourth nodes in the ES cluster is not fully utilized. Therefore, the capacity of the ES cluster can be reduced by deleting at least one fifth target node, thereby effectively improving the resource utilization of the ES cluster and reducing the cost of the ES cluster.

[0036] In a second aspect, a resource scheduling device for an ES cluster is provided. The resource scheduling device may include at least one module. The at least one module may be used to implement the resource scheduling method for the ES cluster provided in the above aspect.

[0037] In a third aspect, a computing device cluster is provided, the computing device cluster includes at least one computing device, each computing device includes a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the resource scheduling method of the ES cluster provided by the above aspect.

[0038] In a fourth aspect, a computer-readable storage medium is provided, in which computer program instructions are stored. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the resource scheduling method for the ES cluster provided in the above aspect.

[0039] In a fifth aspect, a computer program product including instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the resource scheduling method for the ES cluster provided in the above aspects.

[0040] In a sixth aspect, an ES system is provided, which includes: an ES cluster, and a resource scheduling device for the ES cluster, and the resource scheduling device can be used to implement the resource scheduling method for the ES cluster provided in the above aspects.

[0041] In summary, the present application provides a resource scheduling method, device and system for an elastic search cluster, which belongs to the field of data storage technology. The solution provided by the present application can count the data write rate of the target index, and can dynamically adjust the amount of resources of the nodes occupied by multiple shards in the target index when the data write rate is outside the target rate range. Since the data storage capacity of the target index and the maximum data write rate supported are related to the amount of resources occupied by the multiple shards, the solution provided by the present application can realize dynamic adjustment of the data storage capacity and maximum data write rate of the target index without changing the number of shards included in the target index. As a result, not only the flexibility of data writing is effectively improved, but also the utilization rate of cluster resources is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the structure of an ES cluster provided in an embodiment of the present application;

[0043] Figure 2 It is a schematic diagram of an application scenario of a resource scheduling method for an ES cluster provided in an embodiment of the present application;

[0044] Figure 3 This is a flow chart of a resource scheduling method for an ES cluster provided in an embodiment of the present application;

[0045] Figure 4 This is a flow chart of a method for expanding the capacity of an ES cluster provided in an embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of adding a second node to an AASS component provided in an embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of a client proxy component migration shard provided in an embodiment of the present application;

[0048] Figure 7 is a schematic diagram of an expanded ES cluster provided in an embodiment of the present application;

[0049] Figure 8 This is a flow chart of a method for reducing the capacity of an ES cluster provided in an embodiment of the present application;

[0050] Fig. 9 It is a schematic diagram of an AASS component instructing a client proxy component to start a deletion process provided by an embodiment of the present application;

[0051] Fig.10 It is a schematic diagram of another client proxy component migration shard provided in an embodiment of the present application;

[0052] Fig.11 is a schematic diagram of an AASS component deleting a first target node provided by an embodiment of the present application;

[0053] Fig.12 is a schematic diagram of data writing performance of a first node with different node specifications provided by an embodiment of the present application;

[0054] Fig.13 It is a flowchart of another ES cluster resource scheduling method provided in an embodiment of the present application;

[0055] Fig.14 It is a schematic diagram of a resource scheduling device determining a node replacement strategy provided by an embodiment of the present application;

[0056] Fig.15 It is a schematic diagram of adding a third node to an AASS component provided in an embodiment of the present application;

[0057] Fig.16 is a schematic diagram of an AASS component deleting a fourth target node provided by an embodiment of the present application;

[0058] Fig.17 It is a schematic diagram of an AASS component provided in an embodiment of the present application that adds a fifth node;

[0059] Fig.18 is a schematic diagram of a data reading frequency of a target index provided in an embodiment of the present application;

[0060] Fig.19 is a schematic diagram of a data writing frequency of a target index provided in an embodiment of the present application;

[0061] Fig. 20 It is a structural diagram of a resource scheduling device of an ES cluster provided in an embodiment of the present application;

[0062] Fig.21 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0063] Fig. 22 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following describes in detail the ES cluster resource scheduling method, device and system provided by the embodiments of the present application in conjunction with the accompanying drawings.

[0065] First, the following terms involved in the embodiments of the present application are explained.

[0066] ES: A distributed, highly scalable and real-time data search engine developed based on Lucene (a full-text search engine), which can provide data search services on a cluster basis.

[0067] ES cluster, a collection of multiple nodes, can independently provide search services. For example, refer to Figure 1 , the ES cluster can include 3 nodes, from node 1 to node 3.

[0068] Node: Also called ES server, ES instance or computing instance, it contains basic physical resources such as processor, memory and disk. A node can be a physical machine, a VM, or a container in a physical machine.

[0069] Index: A logical space used to store data, equivalent to a database.

[0070] Shard: obtained by index partitioning, used to carry the data in the index. Each index can be divided into multiple shards, which can be distributed in different nodes, thereby effectively improving the data storage capacity of the index. Each shard can be a Lucene instance. For example, refer to Figure 1 , the index can be divided into 5 shards from shard S1 to shard S5. Among them, shard S1 and shard S3 are distributed in node 1, shard S2 and shard S4 are distributed in node 2, and shard S5 is distributed in node 3.

[0071] Replica: A copy of a shard is a backup of the shard. Each shard can have one or more replicas, and each replica and its corresponding shard are distributed in different nodes. For example, refer to Figure 1 , each shard from shard S1 to shard S5 has one replica, and the replicas of shards S1 to shard S5 are replicas R1 to R5, respectively. Among them, replicas R1 and R3 are distributed in node 3, replicas R2 and R4 are distributed in node 1, and replica R5 is distributed in node 2.

[0072] Since the data stored in the replica is the same as the data stored in the shard, and the replica can process data query requests, that is, provide data query services, it can effectively improve data query throughput and achieve high availability. Among them, for the scenario where a replica is created for a shard, the shard can also be called a primary shard, and the replica can also be called a replica shard.

[0073] Figure 2 Schematic diagram of an application scenario of a resource scheduling method for an ES cluster provided in an embodiment of the present application. Figure 2 As shown, the application scenario includes an ES cluster and an application hosting component. In addition, the application scenario can be divided into a management plane and a data plane. Among them, the management plane of the ES cluster may include a master node, a client node, and a request cache. The master node is used to maintain the state of the ES cluster, such as creating and deleting indexes. The client node is mainly responsible for data storage, as well as processing data index requests and data read requests from applications (also called clients). It can be understood that the master node may also have the functions of a client node. The request cache is used to cache data index requests and data read requests. Among them, the data index request is used to request to write data in the index, which can also be called a data write request; the data read request is used to request to read data in the index, which can also be called a data query request.

[0074] The data plane of the ES cluster can include a read cluster and a write cluster. The read cluster includes node N r1 To Node N rN There are N nodes in total, where N is an integer greater than 1. Each node in the read cluster can be distributed with one or more copies of the index, and the read cluster can be used to process data read requests. The write cluster includes nodes N w1 To Node N wM There are M nodes in total, where M is an integer greater than 1. Each node in the write cluster can be distributed with one or more shards of the index, and the write cluster can be used to process data write requests. That is, when data needs to be written to a certain index, data can be written only to the shards distributed in the write cluster, and the shards in the write cluster synchronize the data to one or more replicas in the read cluster. When reading data from a certain index (i.e., querying data), data can be read only from the replicas distributed in the read cluster. In this way, data read and write separation can be achieved, thereby effectively improving the data read and write efficiency.

[0075] The application hosting component can also be called the smart cloud APP elastic engine (Scase) service component, which can provide global application hosting services. Figure 2 The application hosting components may include: global scheduling components and application queue management components located on the management plane, as well as client proxy components, application gateway (AGW), application autoscale service (AASS) components and server software development kit (SDK) located on the data plane. Figure 2 In the application scenario shown, each shard in the write cluster and each replica in the read cluster can be hosted by a search process (also called a Lucene process or application process). The search process is integrated with the service SDK. The functions of each component in the application hosting component are as follows:

[0076] Service SDK: used to control the start and stop of the search process and report the monitoring data of the search service. The monitoring data may include: the read and write volume per second, the status of the search process, and the number of threads included in the search process.

[0077] Application queue management component: responsible for hosting the search process and managing the size of the search process queue.

[0078] AGW: used to control the search process. For example, it aggregates the monitoring data collected by the service SDK and sends control instructions to the service SDK so that the service SDK can control the start and stop of the search process and control the migration of shards or replicas.

[0079] AASS component: monitors the traffic of the search process (including the amount of written data and the number of data read requests), and coordinates application indicators with the underlying resources of the cluster to implement application-driven elastic services.

[0080] Client proxy component: used to control the flow of data write requests and data read requests, that is, to flow data write requests to each node in the write cluster, and to flow data read requests to each node in the read cluster.

[0081] Global Scheduling Component: Provides a global resource view of the resource pool, and decides how to combine nodes in the ES cluster according to preset policies (such as cost-first policies) based on the node specifications of the available nodes in the resource pool.

[0082] Continue to refer Figure 2 The application scenario may also include a storage service component, such as an object storage service (OBS) component. The storage service component is used to synchronize data between shards and replicas.

[0083] An embodiment of the present application provides a resource scheduling method for an ES cluster, wherein the ES cluster includes multiple first nodes, and multiple shards of a target index are distributed in the multiple first nodes. Among them, at least one shard of the target index can be distributed in each first node. It can be understood that the ES cluster can carry at least one index, and the target index can be any index in the at least one index. It can also be understood that the multiple shards of the target index all refer to primary shards, and the multiple first nodes can form a write cluster of the target index. For example, the multiple first nodes can be Figure 2 Node N in w1 To Node N wM .

[0084] The method provided in the embodiment of the present application can be applied to a resource scheduling device, which can be such as Figure 2 Application hosting components in the application scenario shown. Figure 3 , the resource scheduling method includes:

[0085] Step 101: Count the data writing rate of the target index.

[0086] In an embodiment of the present application, the resource scheduling device may count the data write rate of the target index in real time or periodically. The data write rate counted by the resource scheduling device at each statistical moment may refer to: the average data write rate within the first target duration before the statistical moment, that is, the ratio of the total amount of data received to be written to the target index within the first target duration to the first target duration. The unit of the data write rate may be bits per second (bps).

[0087] Optionally, the resource scheduling device may count the number of data write requests received for the target index within the first target duration, and count the amount of data requested to be written by each data write request. The resource scheduling device may accumulate the amount of data requested by each data write request received within the first target duration, thereby obtaining the total amount of data received within the first target duration and required to be written to the target index. Thereafter, the resource scheduling device may calculate the ratio of the total amount of data to the first target duration, and determine the ratio as the data write rate of the target index.

[0088] For example, it is assumed that the resource scheduling device is Figure 2 The AASS component can calculate the data write rate of the target index based on the data counted by the AGW or client proxy component.

[0089] Step 102: Based on the data write rate of the target index being outside the target rate range, adjust the amount of resources of the nodes occupied by the multiple shards.

[0090] The resource scheduling device pre-stores a target rate range, which may be a range of data write rates that the target index can currently support. Since the target index is currently carried by multiple first nodes, the target rate range may also be understood as a range of data write rates that the multiple first nodes can support. Furthermore, the target rate range may be determined based on the node specifications of the multiple first nodes.

[0091] Among them, the measurement index of the node specification may at least include: the number of processor cores included in the node. Alternatively, the measurement index may also include the memory size of the node. If a node has x processor cores and y gigabytes (G) of memory, the node specification of the node can be expressed as xUyG. Wherein, x and y are both positive integers. U is the abbreviation of unit, which is a unit representing the external size of a server, and can be used to represent the number of processor cores in the embodiment of the present application.

[0092] It is understandable that the higher the node specification of the first node, the higher the maximum data writing rate it can support, that is, the better the data writing performance of the first node. Therefore, the resource scheduling device can determine the target rate range based on the node specifications of the multiple first nodes.

[0093] Optionally, the resource scheduling device may obtain a preset data write rate threshold s0 for each processor core, that is, a single-core data write rate. Afterwards, the target rate range may be determined based on the product of the data write rate threshold s0 and the total number m of processor cores included in the plurality of first nodes. For example, the target rate range may be (s0×m)±Δs. Wherein, Δs is a preset write rate fluctuation value, which may be a number greater than or equal to 0. Wherein, the data write rate threshold s0 may be configured by an application (i.e., a client), and may be obtained by testing the data write performance of a single processor core by the application.

[0094] For example, if the amount of resources required for a single search process of an application to run is 1U4G, and when the single search process runs in a 1U4G node, the maximum data write rate supported by the node is 2 megabits per second (Mbps), then the application can configure the data write rate threshold s0 of each processor core to be 2Mbps. If the amount of resources required for a single search process of an application to run is 2U8G, and when the single search process runs in a 2U8G node, the maximum data write rate supported by the node is 2Mbps, then the application can configure the data write rate threshold s0 of each processor core to be 1Mbps.

[0095] Assume that the data write rate threshold s0 of each processor core configured in the resource scheduling device is 2Mbps. The ES cluster includes 10 first nodes, and the number of processor cores included in each first node is 4, that is, the total number m of processor cores included in the 10 first nodes is 40. Accordingly, the resource scheduling device can determine the target rate range to be (80±Δs)Mbps. If the preset write rate fluctuation value Δs is 0, the target rate range can be equal to 80Mbps.

[0096] When the data write rate of the target index is outside the target rate range, the resource scheduling device can determine that the data write performance of the multiple first nodes does not match the data write rate of the target index. If the data write rate of the target index is greater than the upper limit of the target rate range, the resource scheduling device can determine that the data write performance of the multiple first nodes cannot support the current actual data write rate of the target index, that is, data may accumulate and cannot be written to the target index in a timely and effective manner. If the data write rate of the target index is less than the lower limit of the target rate range, the resource scheduling device can determine that the data write performance of the multiple first nodes is not fully utilized, that is, the resource utilization of the ES cluster is low.

[0097] Based on this, the resource scheduling device can dynamically adjust the amount of resources occupied by multiple shards of the target index so that the amount of resources occupied by the multiple shards can match the actual data writing rate of the target index. In this way, the resource utilization of the ES cluster can be effectively improved while ensuring high data writing performance, that is, a flexible strategy based on write service quality (quality of service, QOS) is realized.

[0098] In a first optional implementation, the resource scheduling device may adjust the number of nodes occupied by the multiple shards in the target index to adjust the amount of resources occupied by the multiple shards. In a second optional implementation, the resource scheduling device may adjust the node specification of at least one first node among the multiple first nodes to adjust the amount of resources occupied by the multiple shards.

[0099] The following describes the implementation process of step 102 by taking the first implementation mode as an example. Figure 4 , the step 102 may include:

[0100] Step 102a1: based on the data write rate of the target index being greater than the upper limit of the target rate range, at least one second node is added to the ES cluster.

[0101] When the data write rate of the target index is greater than the upper limit of the target rate range, in order to increase the amount of resources occupied by the target index, the resource scheduling device can add at least one second node in the ES cluster for carrying the shards of the target index.

[0102] As a first example, the resource scheduling device can determine the number of at least one second node to be added based on the difference between the data write rate of the target index and the upper limit of the target rate range. The number of the at least one second node is positively correlated with the difference. Moreover, the node specifications of the at least one second node are all preset specifications, that is, the node specifications of the at least one second node are the same. Since in the first example, the node specifications of the nodes to be added are all preset specifications, the resource scheduling device can directly determine the number of at least one second node to be added based on the difference in write rates. This method of adding a second node is relatively simple and efficient.

[0103] In the first example, assuming that the number of processor cores in the preset specification is m0, the number n1 of the at least one second node to be added can satisfy:

[0104] n1≥(s1-s2) / (s0×m0) formula (1);

[0105] Wherein, s1 is the data writing rate of the target index, s2 is the upper limit of the target rate range, and s0 is the data writing rate threshold of each processor core. The above formula (1) can also be transformed into: (n1×m0)≥(s1-s2) / s0, where (n1×m0) is the total number of processor cores included in the at least one second node.

[0106] As a second example, if the available nodes in a resource pool (e.g., a cloud platform) have multiple different alternative node specifications, the resource scheduling device can determine the number of at least one second node to be added and the node specification of each second node based on the difference between the data write rate of the target index and the upper limit of the target rate range, and the multiple alternative node specifications. The node specification of each second node is selected from the multiple alternative node specifications, and the node specifications of different second nodes can be the same or different. In this second example, the total number m1 of processor cores included in the at least one newly added second node can satisfy:

[0107] m1≥(s1-s2) / s0 formula (2).

[0108] Optionally, the resource scheduling device can determine the number of at least one second node and the node specification of each second node based on the cost of each alternative node specification, using a cost-first strategy. Thus, the cost of adding new nodes can be effectively reduced while improving the data writing performance of the ES cluster.

[0109] For example, the resource scheduling device may first determine multiple node combinations that can satisfy the above formula (2) from the available nodes in the resource pool, wherein the number of second nodes in different node combinations is different, and / or the specifications of at least one second node are different. Afterwards, the resource scheduling device may select a node combination with the lowest cost from the multiple node combinations, and add at least one second node according to the node combination with the lowest cost.

[0110] Assume that the data write rate s1 of the target index is 100Mbps, the upper limit s2 of the target rate range is 80Mbps, and the data write rate threshold s0 of each processor core is 2Mbps. Based on the above formula (2), the resource scheduling device can determine that the total number m1 of processor cores included in at least one second node to be added must be greater than or equal to 10. If the node specifications of the second nodes to be added are all preset specifications: 2U8G, that is, the number m0 of processor cores in each second node is equal to 2, then the resource scheduling device can add 5 2U8G second nodes in the ES cluster.

[0111] If the available nodes in the resource pool have two alternative node specifications, 2U8G and 4U16G, and the cost of one 4U16G node is lower than the cost of two 2U8G nodes, then the resource scheduling device can add two 4U16G second nodes and one 2U8G second node to the ES cluster based on the cost priority strategy.

[0112] Optionally, if the available nodes in the resource pool include nodes that use spot resources, since the cost of the spot resources is relatively low, in the above step 102a1, the resource scheduling device may preferentially select the node that uses the spot resources as the second node to be added. That is, the resources used by the at least one second node may be spot resources. The node that uses the spot resources may also be called a spot instance.

[0113] Step 102a2: Migrate at least one shard among the multiple shards to the at least one second node.

[0114] After adding at least one second node in the ES cluster, the resource scheduling device can migrate at least one shard in the target index to the at least one second node, wherein each second node can carry one or more shards of the target index.

[0115] Optionally, at least one shard migrated by the resource scheduling device may satisfy at least one of the following conditions: the amount of data written in the shard is less than a first threshold; the total number of shards distributed in the first node where the shard is located is greater than a second threshold.

[0116] Based on the above conditions, it can be known that the resource scheduling device can select the shards with a smaller amount of written data for migration, and / or select the shards in the first node with a larger number of shards for migration. In this way, the migration efficiency of the shards can be effectively improved, the migration cost can be reduced, and the number of shards in each first node can be ensured to be relatively balanced.

[0117] The first threshold and the second threshold may both be fixed values ​​pre-configured in the resource scheduling device. Alternatively, the first threshold may be determined based on the amount of data written to each shard in the target index, for example, it may be the mean or lower quartile of the amount of data written to each shard of the target index. The second threshold may be determined based on the number of shards in each first node, for example, it may be the mean or upper quartile of the number of shards in each first node.

[0118] It is understandable that the resource scheduling device may also determine at least one shard to be migrated in other ways. For example, the resource scheduling device may randomly select at least one shard to be migrated from multiple shards of the target index.

[0119] It can also be understood that after determining at least one shard to be migrated, the resource scheduling device can migrate the at least one shard to at least one newly added second node based on a load balancing strategy.

[0120] In the following, the resource scheduling device is used as Figure 2 Taking the application hosting component shown as an example, the implementation process of the above steps 102a1 and 102a2 is introduced.

[0121] First, after detecting that the data write rate of the target index is greater than the upper limit of the target rate range, the AASS component can determine the capacity expansion strategy, which includes the number of at least one second node to be added and the node specifications of each second node to be added. For example, the AASS component can determine the capacity expansion strategy based on the difference between the data write rate and the upper limit of the target rate range. Alternatively, the AASS component can report the difference to the global scheduling component, which can then determine the capacity expansion strategy based on the difference and the multiple alternative node specifications in the resource pool, and send the capacity expansion strategy to the AASS component.

[0122] Afterwards, the AASS component can add at least one second node in the ES cluster according to the expansion strategy and adjust the shard deployment strategy. The shard deployment strategy may refer to the correspondence between each shard in the target index and the nodes to which it is distributed, that is, the mapping relationship between shards and nodes.

[0123] For example, refer to Figure 5, assuming that the target index includes 4 shards from shard S1 to shard S4, and the read cluster used to carry the 4 shards includes 2 first nodes, node 1 and node 2. Among them, shard S1 and shard S3 are distributed in node 1, and shard S2 and shard S4 are distributed in node 2. If the AASS component determines that the data write rate of the target index is greater than the upper limit of the target rate range, it can add 2 second nodes, node 3 and node 4, in the read cluster. In addition, the AASS component can adjust the deployment strategy of the shards to: shard S3 is deployed to node 3, and shard S4 is deployed to node 4.

[0124] Furthermore, the client proxy component can connect to the AASS component in real time to obtain the service status of the newly added second node and synchronize the deployment strategy of the shards. After the client proxy component determines that the service status of the newly added second nodes is in the ready state, it can migrate at least one shard to the at least one newly added second node according to the deployment strategy of the shards. For example, refer to Figure 6 , the client proxy component can migrate shard S3 to node 3 and shard S4 to node 4.

[0125] After completing the shard migration, the client proxy component also needs to update the diversion strategy and divert the data write request according to the updated diversion strategy. The diversion strategy refers to the strategy for distributing data write requests to each node that hosts the target index. For example, refer to Figure 6 , before migrating shards S3 and S4, the client proxy component can distribute data write requests to nodes 1 and 2 based on the diversion strategy. Figure 6 It can be seen that during the migration of shards S3 and S4, the client proxy component does not distribute data write requests to nodes 3 and 4. Figure 7 ,After completing the migration of shards S3 and S4, the client proxy component can distribute data write requests to nodes 1, 2, 3, and 4 based on the updated diversion strategy.

[0126] In the above expansion process, the time required to add at least one second node is about 3 to 5 minutes, and the time required for shard migration and traffic switching is about 1 minute.

[0127] Figure 8 This is a flow chart of a method for reducing the capacity of an ES cluster provided in an embodiment of the present application. Figure 8 , the above step 102 may further include:

[0128] Step 102b1: Based on the data write rate of the target index being less than the lower limit of the target rate range, at least one first target node to be deleted is determined from multiple first nodes.

[0129] When the data write rate of the target index is less than the lower limit of the target rate range, in order to improve the resource utilization of the ES cluster and reduce the cost of the ES cluster, the resource scheduling device can determine at least one first target node to be deleted from multiple first nodes.

[0130] Optionally, the total number of shards distributed in each first target node may be less than a third threshold. That is, the resource scheduling device may delete the first node with a smaller number of shards. Thus, it can be ensured that before deleting the at least one first target node, the shards in the at least one first target node can be quickly migrated to other first nodes.

[0131] The third threshold may be a fixed value preconfigured in the resource scheduling device. Alternatively, the third threshold may be determined based on the number of shards in each first node, for example, the mean or lower quartile of the number of shards in each first node.

[0132] As a first example, if the node specifications of each first node in the ES cluster are preset specifications, the resource scheduling device can determine the number of at least one first target node to be deleted based on the difference between the data write rate of the target index and the lower limit of the target rate range. The number of the at least one first target node is positively correlated with the difference.

[0133] Assuming that the number of processor cores in the preset specification is m0, the number n2 of the at least one first target node to be deleted can satisfy:

[0134] n2≤(s3-s1) / (s0×m0) formula (3);

[0135] Wherein, s1 is the data writing rate of the target index, s3 is the lower limit of the target rate range, and s0 is the data writing rate threshold of each processor core. The above formula (3) can also be transformed into: (n2×m0)≤(s3-s1) / s0, where (n2×m0) is the total number of processor cores included in the at least one first target node.

[0136] It can be understood that the lower limit s3 of the target rate range may be less than the upper limit s2 of the target rate range, or the lower limit s3 of the target rate range may be equal to the upper limit s2, that is, the target rate range may be a rate threshold.

[0137] As a second example, if the node specifications of the first nodes are not exactly the same, the resource scheduling device may determine at least one first target node to be deleted based on the difference between the data write rate of the target index and the lower limit of the target rate range, and the node specifications of the first nodes. In addition, the total number m2 of processor cores included in the at least one first target node may satisfy:

[0138] m2≤(s3-s1) / s0 formula (4).

[0139] Optionally, in the second example, the resource scheduling device may also adopt a cost-first strategy to determine at least one first target node to be deleted based on the costs of first nodes of different node specifications, so as to effectively reduce the cost of the ES cluster.

[0140] For example, the resource scheduling device may first determine a plurality of node combinations that can satisfy the above formula (4), wherein the number of first nodes in different node combinations is different, and / or the specifications of at least one first node are different. Afterwards, the resource scheduling device may select a node combination with the highest cost from the plurality of node combinations, and determine the node combination with the highest cost as the combination of at least one first target node to be deleted.

[0141] Step 102b2: Migrate all shards distributed in at least one first target node to other first nodes except the at least one first target node.

[0142] After the resource scheduling device determines at least one first target node to be deleted, the shards distributed in the at least one first target node can be migrated to other first nodes except the at least one first target node. For example, the resource scheduling device can migrate shards based on a load balancing strategy to ensure that after the shards are migrated, the number of shards in each first node except the at least one first target node is relatively balanced.

[0143] Step 102b3: Delete at least one first target node.

[0144] After the resource scheduling device completes the shard migration, the at least one first target node can be deleted. Thus, the capacity of the ES cluster can be effectively reduced, that is, the capacity of the ES cluster can be reduced, thereby effectively reducing the cost of the ES cluster.

[0145] In the following, the resource scheduling device is used as Figure 2 Taking the application hosting component shown as an example, the implementation process of the above steps 102b1 to 102b3 is introduced.

[0146] First, after detecting that the data write rate of the target index is less than the lower limit of the target rate range, the AASS component can determine a capacity reduction strategy, which includes at least one first target node to be deleted and a deployment strategy for the shards in the at least one first target node. Fig. 9As shown, the AASS component can send a node deletion notification to the client proxy component to instruct the client proxy component to start the deletion process. After receiving the node deletion notification, the client proxy component can synchronize the capacity reduction strategy in the AASS component and migrate all shards in at least one first target node to other first nodes according to the capacity reduction strategy. For example, refer to Fig.10 , the client proxy component can migrate shard S3 in node 3 to node 1, and migrate shard S4 in node 4 to node 2.

[0147] After the shard migration is completed, the client proxy component can cut off the service flow of node 3 and node 4. In addition, the client proxy component can notify the AASS component that the at least one first target node can be deleted. The AASS component can then monitor the service status of the at least one first target node through the AGW. After determining that the data and processes in the at least one first target node are deleted, the AASS component can delete the at least one first target node. For example, refer to Fig.10 and Fig.11 , the AASS component can delete nodes 3 and 4.

[0148] It is understandable that in an embodiment of the present application, the AASS component can directly call the interface of the elastic computing service (ECS) or the autoscaling service (AS) of the ES cluster to add a second node, or delete the first target node. Among them, since the process of calling the ECS interface to delete the node supports the execution of the deletion script, the ECS will execute the deletion callback preset in the service SDK to achieve the deletion of data and processes in the first target node. The AGW can update the service status of the node through the service SDK, and the service status includes the process status and the number of sessions in the node. After the AASS component monitors the AGW and determines that the data and processes in the node are deleted, it will call the ECS interface to automatically delete the first target node. Alternatively, the AGW can also call the deletion function to delete the data and processes in the first target node. When the data and processes in the node are deleted, the ECS interface is called to delete the first target node.

[0149] The following describes the implementation process of step 102 by taking the second implementation method (ie, adjusting the node specification of at least one first node among the multiple first nodes occupied by the target index) as an example.

[0150] In the second implementation, multiple first nodes in the ES cluster can all adopt a serverless architecture. When the resource scheduling device detects that the data write rate of the target index is outside the target rate range, it can adjust the node specification of at least one of the multiple first nodes. That is, the resource scheduling device does not need to adjust the number of nodes used to carry the target index, nor does it need to migrate the shards of the target index, but can dynamically adjust the node specifications of at least one first node to achieve the adjustment of the amount of resources occupied by the target index.

[0151] As mentioned above, the measurement index of the node specification may at least include: the number of processor cores of the node. Therefore, the resource scheduling device adjusting the node specification of the second target node may refer to: adjusting the number of processor cores included in the second target node. If the measurement index of the node specification also includes the memory size of the node, then adjusting the node specification of the second target node may refer to: adjusting the number of processor cores included in the second target node, and the memory size.

[0152] In the second implementation, if the resource scheduling device determines that the data write rate of the target index is greater than the upper limit of the target rate range, the node specification of at least one second target node among the multiple first nodes may be increased.

[0153] The total number of shards distributed in each second target node may be greater than a fourth threshold. That is, the resource scheduling device may increase the node specification of the first node with a larger number of shards. The fourth threshold may be a fixed value pre-configured in the resource scheduling device. Alternatively, the fourth threshold may be determined based on the number of shards in each first node, for example, the mean or upper quartile of the number of shards in each first node.

[0154] It is understandable that the at least one second target node can also be determined in other ways, for example, the resource scheduling device can randomly select at least one second target node from multiple first nodes. Alternatively, the resource scheduling device can determine all of the multiple first nodes as second target nodes, that is, the resource scheduling device can increase the node specification of each first node.

[0155] In the second implementation, if the resource scheduling device determines that the data write rate of the target index is less than the lower limit of the target rate range, the node specification of at least one third target node among the multiple first nodes may be reduced.

[0156] The total number of shards distributed in each third target node may be less than the fifth threshold. That is, the resource scheduling device may reduce the node specification of the first node with a smaller number of shards. The fifth threshold may be a fixed value pre-configured in the resource scheduling device. Alternatively, the fifth threshold may be determined based on the number of shards in each first node, for example, the mean or lower quartile of the number of shards in each first node.

[0157] It is understandable that the at least one third target node can also be determined in other ways, for example, the resource scheduling device can randomly select at least one third target node from multiple first nodes. Alternatively, the resource scheduling device can determine all of the multiple first nodes as the third target node, that is, the resource scheduling device can reduce the node specification of each first node.

[0158] It can also be understood that the resource scheduling device can adjust the node specification of the at least one second target node (or at least one third target node) based on the data write rate s1 of the target index. And, the total number m3 of processor cores included in the plurality of first nodes after adjusting the node specifications can satisfy: m3≥s1 / s0 formula (5).

[0159] For example, assume that the data write rate threshold s0 of each processor core is 2Mbps, and the ES cluster includes 10 first nodes with a node specification of 4U16G, that is, the total number of processor cores included in the 10 first nodes is 40. If the data write rate s1 of the target index is 100Mbps, the resource scheduling device can determine based on the above formula (5) that the total number of processor cores included in the 10 first nodes must be greater than or equal to 50 to meet the data write rate s1 requirement of the target index. Based on this, the resource scheduling device can adjust the node specification of the three first nodes in the ES cluster from 4U16G to 8U32G. At this time, the total number m3 of processor cores included in the multiple first nodes after adjusting the node specification is 52.

[0160] If the data write rate s1 of the target index is 40Mbps, the resource scheduling device can determine based on the above formula (5) that the total number of processor cores included in the 10 first nodes is greater than or equal to 20 to meet the data write rate s1 requirement of the target index. Based on this, the resource scheduling device can adjust the node specifications of the 10 first nodes in the ES cluster from 4U16G to 2U8G. At this time, the total number m3 of processor cores included in the multiple first nodes after the node specifications are adjusted is 20.

[0161] Optionally, when the resource scheduling device detects that the data write rate of the target index is outside the target rate range, it can use any one of the first and second implementations to adjust the amount of resources occupied by multiple shards in the target index. Alternatively, the resource scheduling device can also combine the above two implementations to adjust the amount of resources occupied by multiple shards in the target index, that is, the resource scheduling device can adjust both the number of nodes used to carry the target index and the node specifications of at least one first node.

[0162] In an embodiment of the present application, a correspondence between node specifications and node performance may be pre-configured in the resource scheduling device, and the node performance corresponding to different node specifications is recorded in the correspondence. Among them, the node performance can be characterized by the following parameters: the number of shards that the node can carry, and the maximum data write rate supported by the node. When the resource scheduling device detects that the data write rate of the target index is outside the target rate range, it can determine the node specification of the second node to be added based on the correspondence, and / or adjust the node specification of the first node.

[0163] It is understandable that for search processes (including indexing processes and reading processes) of different types of applications, the amount of resources required for a node to run the search process is different. For example, the amount of resources required to run a single search process of application 1 is 1U2G, and the amount of resources required to run a single search process of application 2 may be 2U4G. Based on this, for each type of application, a corresponding relationship associated with the type of application may be configured in the resource scheduling device, and the node performance in the corresponding relationship may be obtained by testing the type of application.

[0164] For example, the corresponding relationship associated with a certain type of application configured in the resource scheduling device can be shown in Table 1. Referring to Table 1, the number of shards that a node with a node specification of 2U8G can carry is 1, and the maximum data write rate it supports is 4Mbps. The number of shards that a node with a node specification of 8U32G can carry is also 1, but the maximum data write rate it supports can reach 16Mbps.

[0165] Table 1

[0166] Node Specifications The number of shards carried Maximum data writing rate (Mbps) 2U8G 1 4 4U16G 1 8 8U32G 1 16 12U48G 1 24 16U64G 2 32 24U96G 2 48 32U128G 2 64

[0167] refer to Fig.12, assuming that the resource scheduling device adjusts the node specifications of node 1 and node 2 in the ES cluster to 32U, adds node 3 and node 4 with a node specification of 4U in the ES cluster, and migrates shard S3 to node 3 and shard S4 to node 4. Combined with the above Table 1, it can be seen that the maximum data write rate supported by node 1 and node 2 is 64Mbps, and the maximum data write rate supported by node 3 and node 4 is 8Mbps. The total data write rate supported by the four nodes can reach 144Mbps.

[0168] Optionally, the resource used by the fourth target node among the plurality of first nodes may be a bidding resource, which is a resource with a relatively low cost and whose cost (ie, price) varies with supply and demand.

[0169] As a first possible example, the resource scheduling device can purchase a fixed-duration bidding resource in a specific period (e.g., at night) as a resource of the fourth target node. When the running time of the bidding resource reaches the fixed time, it will be automatically released, that is, the bidding resource will be recovered by the resource pool.

[0170] As a second possible example, the resource scheduling device can purchase the bidding resource at a preset target price as the resource of the fourth target node. If the target price is higher than the market price of the bidding resource and the resource inventory of the resource pool is sufficient, the ES cluster can continue to occupy the bidding resource. If the target price is lower than the market price of the bidding resource, or the resource inventory of the resource pool is not available, the resource pool will send an early warning notification to the resource scheduling device and automatically release (i.e., reclaim) the bidding resource after a certain period of time.

[0171] Fig.13 is a flow chart of another resource scheduling method provided by an embodiment of the present application. Fig.13 As shown, the resource scheduling method may also include:

[0172] Step 103: based on the remaining available duration of the bidding resource used by the fourth target node being less than the duration threshold, at least one third node is added to the ES cluster.

[0173] When the remaining available time of the bidding resource is less than the time threshold, the resource scheduling device can add at least one third node in the ES cluster (e.g., a write cluster) to migrate the shards in the fourth target node to the at least one third node. The time threshold can be greater than the time required to migrate the shards in the fourth target node. Thus, it can be ensured that the shards in the fourth target node can complete the migration before the bidding resources used by the fourth target node are released, thereby avoiding impact on the business.

[0174] Optionally, the resources used by the at least one third node newly added by the resource scheduling device may include on-demand resources and / or competitive resources. For example, the resource scheduling device may select on-demand resources and / or competitive resources from the resource pool as resources used by the at least one third node according to a cost-first strategy. Among them, on-demand resources refer to resources purchased according to business needs, and their cost is higher than competitive resources.

[0175] In the first example, if the resource scheduling device purchases the bidding resources according to a fixed duration, the resource scheduling device can monitor the remaining available duration of the bidding resources used by the fourth target node in real time. The remaining available duration can be equal to the difference between the fixed duration and the used duration of the bidding resources. When the remaining available duration is less than the duration threshold pre-configured in the resource scheduling device, the resource scheduling device can add at least one third node to the ES cluster.

[0176] For example, assuming that the fixed duration of the bidding resource purchased by the resource scheduling device for the fourth target node is 2 hours, and the duration threshold is 20 minutes, the resource scheduling device can add at least one third node in the ES cluster when the used duration of the bidding resource reaches 1 hour and 40 minutes. Assuming that the target index is used to store live broadcast log data, since the live broadcast log data is generally packaged at a granularity of 5 minutes, uploaded to the ES cluster and indexed, the migration process of the shard is started when the remaining available duration of the bidding resource is 20 minutes. Sufficient migration time is reserved for the shard, which can avoid affecting business performance.

[0177] If the resource scheduling device purchases the bidding resource at the target price, the resource scheduling device can determine that the remaining available time of the bidding resource is less than the time threshold when receiving the warning notification sent by the resource pool, and add at least one third node in the ES cluster. The time threshold can be the reserved time between the resource pool sending the warning notification and automatically releasing the bidding resource.

[0178] It is understandable that when the resource scheduling device adds at least one third node in the ES cluster, it can determine the number of at least one third node to be added and the specifications of each third node based on the data write rate of the target index. For example, the total number of processor cores included in the at least one third node and other first nodes in the ES cluster except the fourth target node can satisfy the above formula (5).

[0179] Step 104: Migrate all shards in the fourth target node to the at least one third node.

[0180] In the embodiment of the present application, after the resource scheduling device adds at least one third node, the shards in the fourth target node can be migrated to the at least one third node. For example, the resource scheduling device can migrate the shards to the at least one third node based on a load balancing strategy. The implementation process of step 104 can refer to the above steps 102a2 and 102b2, which will not be repeated here.

[0181] Step 105: Delete the fourth target node.

[0182] When the shard migration is completed and the remaining available time of the bidding resource used by the fourth target node is zero, the resource scheduling device can delete the fourth target node to release the bidding resource used by the fourth target node. The implementation process of step 105 can refer to the above step 102b3, which will not be repeated here.

[0183] In the following, the resource scheduling device is used as Figure 2 Taking the application hosting component shown as an example, the implementation process of the above steps 103 to 105 is introduced.

[0184] refer to Fig.14 , the AASS component detects the fourth target node (e.g. Fig.14 When the remaining available duration of the bidding resources used by nodes 3 and 4 in the table is less than the duration threshold, a node replacement strategy can be determined, and the node replacement strategy includes the number of at least one third node to be added, and the node specifications of each third node to be added. For example, the AASS component can report the node demand information to the global scheduling component, and the global scheduling component can then determine the node replacement strategy based on the node demand information and multiple alternative node specifications, and send the node replacement strategy to the AASS component. Among them, the node demand information can refer to the total number of processor cores required to be added.

[0185] After that, the AASS component can add at least one third node to the ES cluster according to the node replacement strategy and adjust the shard deployment strategy. Fig.15 , the AASS component can add two third nodes, node 5 and node 6. In addition, the AASS component can adjust the deployment strategy of the shards: shard S3 is deployed to node 5, and shard S4 is deployed to node 6.

[0186] Furthermore, the client proxy component can migrate at least one shard to at least one newly added third node according to the adjusted shard deployment strategy. Fig.15 , the client proxy component can migrate shard S3 in node 3 to node 5, and migrate shard S4 in node 4 to node 6. When the remaining available time of the bidding resources used by nodes 3 and 4 is zero, such as Fig.16As shown, the AASS component can delete the node 3 and the node 4.

[0187] Based on the above analysis, it can be known that the method provided in the embodiment of the present application can achieve flexible adjustment of the data writing performance of the target index by adjusting the amount of resources occupied by multiple shards in the target index, that is, it can achieve the write elasticity of the ES cluster. Based on this, the solution provided in the embodiment of the present application can be used to store a large amount of written data, and the amount of written data fluctuates greatly over time. For example, the target index can be used to store media log data or transaction data of a transaction system. Among them, the media log data may include: live log data and on-demand log data, etc.

[0188] The above is an explanation based on an example in which a resource scheduling device adjusts the amount of resources occupied by multiple shards in a target index according to the data write rate of the target index. In an embodiment of the present application, the ES cluster may also include multiple fourth nodes, and the target index may also include multiple replicas, which are carried by the multiple fourth nodes. The multiple fourth nodes may form a read cluster of the target index. For example, the multiple fourth nodes may be Figure 2 Node N in r1 To Node N rN The resource scheduling device can also flexibly adjust the number of replicas included in the target index based on the data reading frequency of the target index to achieve the read elasticity of the ES cluster. Fig.13 The resource scheduling method provided in the embodiment of the present application may also include:

[0189] Step 106: Count the data reading frequency of the target index.

[0190] In an embodiment of the present application, the resource scheduling device may count the data reading frequency of the target index in real time or periodically, and the data reading frequency may refer to the number of data reading requests for the target index received in a unit time. Optionally, the data reading frequency counted by the resource scheduling device at each statistical moment may refer to: the ratio of the number of data reading requests received within the second target duration before the statistical moment to the second target duration. For example, assuming that the statistical unit of the second target duration is seconds, the data reading frequency may be the number of requests per second (queries persecond, QPS).

[0191] The second target duration may be equal to or different from the first target duration when the statistical data is written into the rate. For example, both the second target duration and the first target duration may be 5 minutes.

[0192] Step 107: Based on the data reading frequency of the target index being outside the target frequency range, adjust the number of replicas included in the target index.

[0193] Each replica in the target index is obtained by copying a shard of the target index, that is, the data stored in each replica is the same as the data stored in the shard corresponding to the replica. In addition, the number of replicas in different shards in the target index can be the same or different.

[0194] It is understandable that the copies of the target index can provide data query services, and the more copies the target index includes, the higher the maximum data reading frequency supported by the target index. In an embodiment of the present application, a target frequency range is pre-stored in the resource scheduling device, and the target frequency range can be determined based on the number of copies currently included in the target index.

[0195] It is also understood that when storing a document in a target index (a document is the smallest unit of data storage), the identifier of the target shard used to store the document needs to be calculated based on the number of shards included in the target index. For example, the identifier shard_num of the target shard can be calculated using the following formula:

[0196] shard_num=hash(routing)%num_primary_shards formula (6);

[0197] Where hash represents a hash operation, routing is a preset routing key, and num_primary_shards is the number of shards included in the target index. When reading the document from the target index, the identifier of the target shard storing the document is also determined based on the above formula (6). Based on the above analysis, it can be seen that in order to ensure accurate data query, after the target index is created, the number of shards included in the target index cannot be modified.

[0198] Since the number of shards included in the target index cannot be adjusted, but the number of replicas can be flexibly adjusted, the more replicas the target index includes, the higher the maximum data reading frequency it can support, that is, the better the data reading performance of the multiple fourth nodes. And since the number of replicas that each fourth node in the ES cluster can carry is positively correlated with the node specification of the fourth node, in the embodiment of the present application, the target frequency range can be determined based on the node specifications of the multiple fourth nodes in the ES cluster.

[0199] The plurality of fourth nodes may intersect with the plurality of first nodes described above, or the plurality of fourth nodes may be independent of the plurality of first nodes. Figure 2 , assuming that the ES cluster is deployed in a read-write separation manner, the multiple first nodes may refer to nodes in the write cluster, and the multiple fourth nodes may refer to nodes in the read cluster.

[0200] Optionally, the resource scheduling device may obtain a preset data reading frequency threshold r0 for each processor core, that is, a single-core data reading frequency. Afterwards, the target frequency range may be determined based on the product of the data reading frequency threshold r0 and the total number m4 of processor cores included in the plurality of fourth nodes. For example, the target frequency range may be (r0×m4)±Δr. Δr is a preset reading rate fluctuation value, which may be a number greater than or equal to 0.

[0201] For example, assuming that the data read frequency threshold r0 of each processor core is 5QPS, and the ES cluster includes 10 4U16G fourth nodes, the total number m4 of processor cores included in the 10 fourth nodes is 40. Accordingly, the resource scheduling device can determine the target frequency range to be (200±Δr)QPS. If the preset read rate fluctuation value Δr is 0, the target frequency range can be equal to 200QPS.

[0202] When the data reading frequency of the target index is outside the target frequency range, the resource scheduling device can determine that the data reading performance of the multiple fourth nodes does not match the data reading frequency of the target index. If the data reading frequency of the target index is greater than the upper limit of the target frequency range, the resource scheduling device can determine that the data reading performance of the multiple fourth nodes cannot support the data reading frequency. If the data reading frequency of the target index is less than the lower limit of the target frequency range, the resource scheduling device can determine that the data reading performance of the multiple fourth nodes is not fully utilized, that is, the resource utilization of the ES cluster is low.

[0203] Based on this, the resource scheduling device can adjust the number of replicas included in the target index so that the number of replicas included in the target index can match the data reading frequency. Thus, the resource utilization of the ES cluster can be effectively improved while ensuring data reading performance.

[0204] In an embodiment of the present application, if the data reading frequency of the target index is greater than the upper limit of the target frequency range, the resource scheduling device may add at least one fifth node in the ES cluster, and may add at least one copy of the target index in each fifth node. The number of newly added copies in each fifth node may be positively correlated with the node specification of the fifth node. That is, the higher the node specification of the fifth node, the more newly added copies may be in the fifth node.

[0205] As a first possible example, the node specifications of the at least one fifth node are all preset specifications. Accordingly, the number of the at least one fifth node can be determined based on the difference between the data reading frequency and the upper limit of the target frequency range, and the number of the at least one fifth node is positively correlated with the difference.

[0206] In the first example, assuming that the number of processor cores in the preset specification is m0, the number n3 of the at least one fifth node to be added can satisfy:

[0207] n3≥(r1-r2) / (r0×m0) formula (7);

[0208] Wherein, r1 is the data reading frequency of the target index, r2 is the upper limit of the target frequency range, and r0 is the data reading frequency threshold of each processor core. The above formula (7) can also be transformed into: (n3×m0)≥(r1-r2) / r0, where (n3×m0) is the total number of processor cores included in the at least one newly added fifth node.

[0209] Assume that the data read frequency threshold r0 of each processor core is 5QPS, the upper limit r2 of the target frequency range is 200QPS, and the node specifications of the available nodes in the resource pool are all 4U16G, that is, m0 = 4. Fig.17 , if the AASS component detects through the AGW that the data reading frequency r1 of the target index is 210QPS, then the AASS component can determine that a fifth node with a node specification of 4U16G needs to be added according to the above formula (7). Fig.17 As shown, the AASS component can add a new node 3 in the ES cluster and add a new replica in node 3. The replica is hosted by a search process, and the service SDK is integrated in the search process.

[0210] As a second possible example, the available nodes in the resource pool have multiple different alternative node specifications. The resource scheduling device can determine the number of at least one fifth node to be added and the node specification of each fifth node based on the difference between the data reading frequency and the upper limit of the target frequency range, and the multiple different alternative node specifications. The node specification of each fifth node is selected from the specifications of the multiple alternative nodes, and the node specifications of different fifth nodes can be the same or different.

[0211] In the second example, the total number m5 of processor cores included in the at least one fifth node may satisfy:

[0212] m5≥(r1-r2) / r0 formula (8).

[0213] Optionally, the resource scheduling device can determine the number of at least one fifth node and the node specification of each fifth node based on the cost of each alternative node specification, using a cost-first strategy. Thus, the cost of adding new nodes can be reduced while ensuring that the data reading performance of the ES cluster is effectively improved.

[0214] In an embodiment of the present application, if the data reading frequency of the target index is less than the lower limit of the target frequency range, the resource scheduling device can determine at least one fifth target node from multiple fourth nodes, and can delete the copies distributed in each fifth target node. After that, the resource scheduling device can delete the at least one fifth target node.

[0215] Among them, the at least one fifth target node can meet at least one of the following conditions: the number of copies distributed in the node is less than the sixth threshold; the data reading frequency of the node is less than the seventh threshold. That is, the resource scheduling device can delete the fourth node with a small number of shards, and / or delete the fourth node with a small number of received data reading requests. Both the sixth threshold and the seventh threshold can be fixed values ​​pre-configured in the resource scheduling device. Alternatively, the sixth threshold can be determined based on the number of copies in each fourth node, for example, it can be the mean or lower quartile of the number of copies in each fourth node. The seventh threshold can be determined based on the data reading frequency of each fourth node, for example, it can be the mean or lower quartile of the data reading frequency in each fourth node.

[0216] Fig.18 is a schematic diagram of a data reading frequency of a target index provided in an embodiment of the present application, and Fig.18 The target index is used to store media log data as an example for illustration. Fig.18 The horizontal axis is time in hours, and the vertical axis is data reading frequency in times / minute. Fig.18 It can be seen that the data reading frequency of the target index is relatively low overall, with a peak value of about 7 times / minute. If the data reading frequency supported by a fourth node in the ES cluster is 20QPS, good data reading performance can be ensured.

[0217] Fig.19 is a schematic diagram of a data writing frequency of a target index provided in an embodiment of the present application, and Fig.19 The target index is used to store media log data as an example for illustration. Fig.19 The horizontal axis is time in hours, and the vertical axis is data writing frequency in times / minute. Fig.19 It can be seen that the peak range of the data writing frequency of the target index is approximately 5600 (W) times / minute to 13.7W times / minute. Since the data writing frequency of the target index is high, the data writing rate of the target index is also high. Fig.19It can also be seen that the data write rate of the target index fluctuates greatly over time. Since the solution provided by the embodiment of the present application can dynamically adjust the amount of resources occupied by the target index according to the data write rate of the target index, it is possible to achieve dynamic elasticity of data write performance, thereby effectively improving the resource utilization of the ES cluster while ensuring data write performance.

[0218] It is understandable that the order of the steps of the resource scheduling method of the ES cluster provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly. For example, the above steps 103 to 105 can be deleted according to the situation; or, the above steps 106 and 107 can be deleted according to the situation; or, the above steps 106 and 107 can be executed before step 105; or, the above steps 103 to 105 can be executed before step 102.

[0219] In summary, the embodiment of the present application provides a resource scheduling method for an ES cluster. The method can count the data write rate of the target index, and can dynamically adjust the amount of resources of the nodes occupied by multiple shards of the target index when the data write rate is outside the target rate range. Since the data storage capacity of the target index and the maximum data write rate supported are related to the amount of resources occupied by the multiple shards, the method provided by the embodiment of the present application can realize the dynamic adjustment of the data storage capacity and the maximum data write rate of the target index without changing the number of shards included in the target index. Thus, not only the flexibility of data writing is effectively improved, but also the utilization rate of cluster resources is effectively improved.

[0220] In addition, the method provided in the embodiment of the present application can also flexibly adjust the number of copies included in the target index based on the data reading frequency of the target index, thereby realizing the read elasticity of the ES cluster. And because when adding a new node in the ES cluster, the node specifications of the newly added node and the type of resources used by the node can be flexibly selected, the flexibility of adding a new node is effectively improved, and the cost of the newly added node can be reduced. For example, according to testing, in the scenario where the target index stores log data, the cost of the ES cluster can be reduced by 50% by realizing the elasticity of the write cluster. If the resource scheduling device can also flexibly select the node specifications of the node and the type of resources used by the node based on the principle of cost priority, the cost of the ES cluster can be reduced by 70%.

[0221] The embodiment of the present application provides a resource scheduling device for an ES cluster, which can be used to implement the resource scheduling method for an ES cluster provided in the above method embodiment. The ES cluster includes multiple first nodes for carrying a target index, and the target index includes multiple shards. The resource scheduling device can be deployed in a cloud platform, for example, the resource scheduling device can be such as Figure 2 The application hosting components shown in Figure 1. Fig. 20 As shown, the resource scheduling device includes:

[0222] The statistical module 201 is used to count the data writing rate of the target index. The functional implementation of the statistical module 201 can refer to the relevant description of the above step 101.

[0223] The adjustment module 202 is used to adjust the amount of node resources occupied by the multiple shards based on the data write rate of the target index being outside the target rate range. The functional implementation of the adjustment module 202 can refer to the relevant description of the above step 102.

[0224] Optionally, the adjustment module 202 can be used to: add at least one second node in the ES cluster based on the data write rate of the target index being greater than the upper limit of the target rate range; and migrate at least one shard of the multiple shards to the at least one second node. The functional implementation of the adjustment module 202 can also refer to the relevant description of the above steps 102a1 to 102a2.

[0225] As an optional implementation, the adjustment module 202 can also be used to: before adding at least one second node in the ES cluster, determine the number of the at least one second node to be added based on the difference between the data write rate of the target index and the upper limit of the target rate range. The number is positively correlated with the difference, and the node specifications of the at least one second node are all preset specifications.

[0226] As another optional implementation, the adjustment module 202 can also be used to: before adding at least one second node to the ES cluster, determine the number of the at least one second node to be added and the node specification of each second node based on the difference between the data write rate of the target index and the upper limit of the target rate range, and multiple different alternative node specifications. The node specification of each second node is selected from the specifications of the multiple alternative nodes.

[0227] Optionally, the total number m1 of processor cores included in the at least one second node satisfies: m1≥(s1-s2) / s0, where s1 is the data write rate of the target index, s2 is the upper limit of the target rate range, and s0 is the data write rate threshold of each processor core.

[0228] Optionally, the resources used by the at least one second node are bidding resources.

[0229] Optionally, the at least one shard satisfies at least one of the following conditions: the amount of data written in the shard is less than a first threshold; the total number of shards distributed in the first node where the shard is located is greater than a second threshold.

[0230] Optionally, the adjustment module 202 can be used to: determine at least one first target node to be deleted from the multiple first nodes based on the data write rate of the target index being less than the lower limit of the target rate range; migrate all shards distributed in the at least one first target node to other first nodes except the at least one first target node; and delete the at least one first target node. The functional implementation of the adjustment module 202 can also refer to the relevant description of the above steps 102b1 to 102b3.

[0231] Optionally, the at least one first target node satisfies: the total number of shards distributed in the node is less than a third threshold.

[0232] Optionally, the adjustment module 202 may be configured to adjust a node specification of at least one of the plurality of first nodes based on a data write rate of the target index being outside a target rate range.

[0233] Optionally, the adjustment module 202 may be used to: increase the node specification of at least one second target node among the plurality of first nodes based on the data write rate of the target index being greater than the upper limit of the target rate range. Alternatively, reduce the node specification of at least one third target node among the plurality of first nodes based on the data write rate of the target index being less than the lower limit of the target rate range.

[0234] Optionally, the resource used by the fourth target node among the multiple first nodes is a bidding resource; the adjustment module 202 can also be used to: based on the remaining available duration of the bidding resource used by the fourth target node being less than the duration threshold, add at least one third node to the ES cluster; migrate all the shards distributed in the fourth target node to the at least one third node; delete the fourth target node. The functional implementation of the adjustment module 202 can also refer to the relevant descriptions of the above steps 103 to 105.

[0235] Optionally, the resources used by the at least one third node include on-demand resources and / or competitive bidding resources.

[0236] Optionally, the ES cluster may further include multiple fourth nodes, which are used to carry multiple copies of the target index. The statistical module 201 may also be used to count the data read frequency of the target index. The functional implementation of the statistical module 201 may also refer to the relevant description of step 106 above.

[0237] The adjustment module 202 may also be used to adjust the number of replicas included in the target index based on the data reading frequency of the target index being outside the target frequency range. The implementation of the function of the adjustment module 202 may also refer to the relevant description of step 107 above.

[0238] Optionally, the adjustment module 202 can be used to: add at least one fifth node in the ES cluster based on the data reading frequency of the target index being greater than the upper limit of the target frequency range; and add at least one replica of the target index in each fifth node.

[0239] Optionally, the adjustment module 202 can be used to: based on the data reading frequency of the target index being less than the lower limit of the target frequency range, delete the copies distributed in at least one fifth target node among the multiple fourth nodes, and delete the at least one fifth target node.

[0240] It is understandable that if the resource scheduling device is Figure 2 The application hosting components shown, the function of the statistics module 201 can be implemented by the service SDK, AGW and AASS components, and the function of the adjustment module 202 can be implemented by the AASS component, the client agent component and the global scheduling component.

[0241] In summary, an embodiment of the present application provides a resource scheduling device for an ES cluster. The device can count the data write rate of the target index, and can dynamically adjust the amount of resources of the nodes occupied by multiple shards of the target index when the data write rate is outside the target rate range. Since the data storage capacity of the target index and the maximum data write rate supported are related to the amount of resources occupied by the multiple shards, the device provided in the embodiment of the present application can achieve dynamic adjustment of the data storage capacity and the maximum data write rate of the target index without changing the number of shards included in the target index. As a result, not only the flexibility of data writing is effectively improved, but also the utilization rate of cluster resources is effectively improved.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the resource scheduling device and each module described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0243] It is understandable that each module in the resource scheduling device (such as the statistical module 201 and the adjustment module 202) can be implemented by software or by hardware. Exemplarily, the implementation of the statistical module 201 is described below by taking the statistical module 201 as an example. Similarly, the implementation of the adjustment module 202 can refer to the implementation of the statistical module 201.

[0244] As an example of a software functional unit, the statistical module 201 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the statistical module 201 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0245] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0246] As an example of a hardware functional unit, the statistical module 201 may include at least one computing device, such as a server, etc. Alternatively, the statistical module 201 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0247] The multiple computing devices included in the statistical module 201 can be distributed in the same region or in different regions. The multiple computing devices included in the statistical module 201 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the statistical module 201 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0248] It should be noted that, in other embodiments, the statistical module 201 can be used to execute any step in the above resource scheduling method, and the adjustment module 202 can also be used to execute any step in the above resource scheduling method. The steps that the statistical module 201 and the adjustment module 202 are responsible for implementing can be specified as needed, and the statistical module 201 and the adjustment module 202 respectively implement different steps in the resource scheduling method to realize all functions of the resource scheduling device.

[0249] Fig.21 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. Fig.21 , the computing device may include: a bus 2102, a processor 2104, a memory 2106, and a communication interface 2108. The processor 2104, the memory 2106, and the communication interface 2108 communicate through the bus 2102. The computing device may be a server or a terminal device. It should be understood that the embodiments of the present application do not limit the number of processors and memories in the computing device.

[0250] The bus 2102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.21 The bus 2104 may include a path for transmitting information between various components of the computing device (eg, memory 2106, processor 2104, communication interface 2108).

[0251] The processor 2104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0252] The memory 2106 may include a volatile memory, such as a random access memory (RAM). The processor 2104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0253] The memory 2106 stores executable program codes, and the processor 2104 executes the executable program codes to respectively implement the functions of the aforementioned statistical module and the adjustment module, thereby implementing the resource scheduling method provided by the above method embodiment. That is, the memory 2106 stores instructions for executing the resource scheduling method.

[0254] The communication interface 2103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device and other devices or a communication network.

[0255] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0256] like Fig. 22 As shown, the computing device cluster includes at least one computing device. The memory 2106 in one or more computing devices in the computing device cluster may store the same instructions for executing the above resource scheduling method.

[0257] In some possible implementations, the memory 2106 of one or more computing devices in the computing device cluster may also store some instructions for executing the resource scheduling method. In other words, a combination of one or more computing devices may jointly execute instructions for executing the resource scheduling method.

[0258] It should be noted that the memory 2106 in different computing devices in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the resource scheduling device. That is, the instructions stored in the memory 2106 in different computing devices can implement the functions of one or more modules in the statistical module and the adjustment module.

[0259] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network.

[0260] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the resource scheduling method of the ES cluster provided in the above method embodiment.

[0261] The present application embodiment also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the resource scheduling method of the ES cluster provided in the above method embodiment.

[0262] The embodiment of the present application also provides an elastic search system, which can provide a cloud search service (CSS). The elastic search system may include such as Figure 1 The ES cluster shown, and a resource scheduling device for scheduling resources of the ES cluster.

[0263] The ES cluster may include multiple nodes, which may be deployed in a read-write separation manner and a storage-computing separation manner.

[0264] The resource scheduling device can be used to implement the resource scheduling method of the ES cluster provided by the above method embodiment. For example, the resource scheduling device can be Figure 2 The application hosting component shown in FIG. 1 and the structure of the resource scheduling device can refer to Figure 20 to Figure 22 Any of the accompanying drawings.

[0265] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "and / or" in the present application is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0266] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource scheduling method, characterized in that: Applied to an elastic search system, the elastic search system includes a client agent, a write cluster and a read cluster, the write cluster includes a plurality of first nodes, the plurality of first nodes are used to write data to an index of the elastic search system, and the plurality of first nodes are distributed with a plurality of shards of the index, the read cluster includes a plurality of second nodes, the plurality of second nodes are used to read data from an index of the elastic search system, and the plurality of second nodes are distributed with a plurality of replicas corresponding to the plurality of shards of the index; The method comprises: The client agent receives a data write request and a data read request, wherein the data write request is used to write data in the index and the data read request is used to read data in the index; The client agent distributes the data write request to the write cluster, and a first node in the write cluster processes the data write request to write data in one or more shards distributed by the first node; The client agent distributes the data read request to the read cluster, and a second node in the read cluster processes the data read request to read data from one or more replicas distributed by the second node.

2. The method according to claim 1, characterized in that: The amount of resources occupied by one or more shards of the index in the write cluster is adjustable; and / or, The amount of resources occupied by one or more copies of the index in the read cluster is adjustable.

3. The method according to claim 2, characterized in that The amount of resources occupied by one or more shards of the index in the write cluster can be adjusted, including: adjusting the node specification of the first node in the write cluster, the node specification of the first node including the number of processor cores of the first node and / or the memory size of the first node; and / or, The amount of resources occupied by one or more copies of the index in the read cluster can be adjusted, including: adjusting the node specification of the second node in the read cluster, the node specification of the second node including the number of processor cores of the second node and / or the memory size of the second node.

4. The method according to claim 2 or 3, characterized in that: The amount of resources occupied by one or more shards of the index in the write cluster can be adjusted, including: Add one or more first nodes to the write cluster; or, One or more first nodes in the write cluster are deleted.

5. The method according to claim 4, characterized in that The method comprises: After one or more first nodes are added to the write cluster, one or more shards on one or more existing first nodes in the write cluster are migrated to the newly added first nodes; or, Before deleting one or more first nodes in the write cluster, migrate shards on the first nodes to be deleted to other first nodes in the write cluster.

6. The method according to claim 5, characterized in that The strategy for migrating shards in the write cluster is a load balancing strategy.

7. The method according to claim 5 or 6, characterized in that: The step of migrating one or more shards on one or more existing first nodes to a newly added first node in the write cluster includes: Migrate the target shard of the existing first node to the newly added first node, where the target shard satisfies at least one of the following conditions: The amount of data written to the target shard is less than a first threshold; The total number of shards distributed in the existing first node where the target shard is located is greater than a second threshold.

8. The method according to any one of claims 2 to 7, characterized in that: The amount of resources occupied by one or more copies of the index in the read cluster can be adjusted, including: Add one or more second nodes to the read cluster and / or add one or more replicas to the read cluster; or, Deleting one or more of the second nodes in the read cluster and / or deleting one or more replicas in the read cluster.

9. The method according to claim 8, characterized in that The method comprises: After adding one or more second nodes to the read cluster, migrate one or more replicas on one or more existing second nodes to the new second nodes and / or add one or more replicas on the new second nodes in the read cluster; or, Before deleting one or more second nodes in the read cluster, in the read cluster, the replicas on the second nodes to be deleted are migrated to other second nodes or the replicas on the second nodes to be deleted are deleted.

10. The method according to any one of claims 2 to 9, characterized in that: The amount of resources occupied by one or more shards of the index in the write cluster can be adjusted, including: when the data write rate of the index is outside the target rate range, adjusting the amount of resources occupied by one or more shards of the index; and / or, The amount of resources occupied by one or more copies of the index in the read cluster can be adjusted, including: when the data reading frequency of the index is outside a target frequency range, adjusting the amount of resources occupied by one or more copies of the index.

11. An elastic search system, characterized in that: The elastic search system includes a client agent, a write cluster and a read cluster; The write cluster includes a plurality of first nodes, the plurality of first nodes are used to write data to the index of the elastic search system, and the plurality of first nodes are distributed with a plurality of shards of the index; The read cluster includes a plurality of second nodes, the plurality of second nodes are used to read data from the index of the elastic search system, and the plurality of second nodes are distributed with a plurality of replicas corresponding to a plurality of shards of the index; The client agent is used to receive a data write request and a data read request, and to shunt the data write request to the write cluster and shunt the data read request to the read cluster, wherein the data write request is used to write data in the index and the data read request is used to read data in the index; A first node in the write cluster, configured to process the data write request to write data in one or more shards distributed on the first node; The second node in the read cluster is used to process the data read request to read data from one or more replicas distributed by the second node.

12. The elastic search system according to claim 11, characterized in that: The amount of resources occupied by one or more shards of the index in the write cluster is adjustable; and / or, The amount of resources occupied by one or more copies of the index in the read cluster is adjustable.

13. The elastic search system according to claim 12, characterized in that: The elastic search system includes a resource scheduling device, and the resource scheduling device is used to: adjusting the node specification of the first node in the write cluster, where the node specification of the first node includes the number of processor cores of the first node and / or the memory size of the first node; and / or, The node specification of the second node in the read cluster is adjusted, where the node specification of the second node includes the number of processor cores of the second node and / or the memory size of the second node.

14. The elastic search system according to claim 12 or 13, characterized in that: The elastic search system includes a resource scheduling device, and the resource scheduling device is used to: Add one or more first nodes to the write cluster; or, One or more first nodes in the write cluster are deleted.

15. The elastic search system according to claim 14, characterized in that: The resource scheduling device included in the elastic search system is used for: After one or more first nodes are added to the write cluster, one or more shards on one or more existing first nodes in the write cluster are migrated to the newly added first nodes; or, After deleting one or more first nodes in the write cluster, shards on the first nodes to be deleted are migrated to other first nodes in the write cluster.

16. The elastic search system according to claim 15, characterized in that: The strategy for migrating shards in the write cluster is a load balancing strategy.

17. The elastic search system according to claim 15 or 16, characterized in that: The resource scheduling device is used to migrate one or more shards on one or more existing first nodes to a newly added first node in the write cluster, including: Migrate the target shard of the existing first node to the newly added first node, where the target shard satisfies at least one of the following conditions: The amount of data written to the target shard is less than a first threshold; The total number of shards distributed in the existing first node where the target shard is located is greater than a second threshold.

18. The elastic search system according to any one of claims 12 to 17, characterized in that: The elastic search system includes a resource scheduling device, and the resource scheduling device is used to: Add one or more second nodes to the read cluster and / or add one or more replicas to the read cluster; or, Deleting one or more of the second nodes in the read cluster and / or deleting one or more replicas in the read cluster.

19. The elastic search system according to claim 18, characterized in that: The resource scheduling device included in the elastic search system is used for: After adding one or more second nodes to the read cluster, in the read cluster, one or more replicas existing on one or more of the second nodes are migrated to the newly added second nodes and / or one or more replicas are added to the newly added second nodes; or, Before deleting one or more second nodes in the read cluster, in the read cluster, the replicas on the second nodes to be deleted are migrated to other second nodes or the replicas on the second nodes to be deleted are deleted.

20. The elastic search system according to any one of claims 12 to 19, characterized in that: The elastic search system includes a resource scheduling device; The amount of resources occupied by one or more shards of the index in the write cluster can be adjusted, including: when the data write rate of the index is outside the target rate range, the resource scheduling device adjusts the amount of resources occupied by one or more shards of the index; and / or, The amount of resources occupied by one or more copies of the index in the read cluster is adjustable, including: when the data reading frequency of the index is outside the target frequency range, the resource scheduling device adjusts the amount of resources occupied by one or more copies of the index.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The computer program product comprises instructions; when the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 10.

Images