Distributed query method and device, electronic equipment and storage medium
By dynamically selecting the cluster in the distributed query system, the problems of load imbalance and performance instability are solved, query efficiency and response time are improved, and the balancing of the cluster load and resource utilization are achieved.
Patent Information
- Application Number
- CN202411079077.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-06
AI Technical Summary
In a distributed query system, the load imbalance of query requests results in a long wait time for users, and the cluster performance changes over time and is unstable, so it is impossible to update the routing policy in real time to adapt to new situations.
By converting query requests into query tasks and determining the current weights of selected and unselected clusters based on the task model, dynamically selecting the appropriate cluster for query task allocation, ensuring the efficiency and flexibility of query results.
It improves query efficiency, reduces the response time and concurrency of user queries, makes query tasks more efficient when multiple requests, and maintains the balancing of cluster load and resource utilization.
Smart Images

Figure CN119088855A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, and in particular to a distributed query method, device, electronic device and storage medium. Background Art
[0002] An ad hoc query system is a system that provides users with instant data query services. Users can customize query conditions, and the system can return results based on these conditions. It has high requirements for real-time performance and can quickly respond to user query requests and provide real-time data. The data query engine is a very important component of the ad hoc query system. These engines provide efficient, flexible and powerful support for processing and analyzing large-scale data, while supporting distributed queries on multiple data sources. Then, during distributed queries, there are usually problems such as unbalanced load, too many query requests, and long user waiting times. Therefore, how to reasonably distribute query requests to each node and maintain load balancing in the cluster is still a technical problem that needs to be solved. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the present disclosure provides a distributed query method, device, electronic device and storage medium.
[0004] According to a first aspect of an embodiment of the present disclosure, a distributed query method is provided, comprising: in response to obtaining multiple distributed query requests, converting query parameters corresponding to the query requests into query tasks based on a task model; determining a cluster corresponding to the query task based on the query task, the cluster comprising a first cluster and a second cluster, the first cluster being used to process non-scheduled queries, and the second cluster being used to process scheduled queries; determining whether the cluster is a selected cluster or an unselected cluster based on a current weight of the cluster, wherein the current weight of the cluster is used to determine a selected cluster for query tasks other than the first query task; determining a selected cluster corresponding to the query task based on the current weight, and updating the current weights of the selected cluster and the unselected cluster; and polling the updated cluster corresponding to the query task based on the query task to obtain a query result.
[0005] In one embodiment, determining the selected cluster corresponding to the query task based on the current weight includes: in the first query, selecting the cluster with the largest static weight as the selected cluster, performing the query, and updating the current weight, wherein the static weight is a preset value used to determine the selected cluster in the first query, and the current weight is determined based on the static weight and / or the weight change value; in the i+1th query, updating the current weight, wherein the current weight of the selected cluster is the sum of the weight change value in the i-1th query and the static weight of the selected cluster, the current weight of the unselected cluster is the sum of the current weight in the i-1th query and the static weight of the unselected cluster, the weight change value is the difference between the current weight of the selected cluster and the weight sum value, and the weight sum value is the sum of the static weights of the selected cluster and the unselected clusters, judging the current weight of the cluster, and selecting the cluster with the largest current weight as the selected cluster.
[0006] In one implementation, determining the cluster corresponding to the query task based on the query task includes: obtaining multiple query tasks and determining the query parameter identifier of the query task; if the query parameter identifier is a non-scheduled query, the query task corresponds to the first cluster; if the query parameter identifier is a scheduled query, the query task corresponds to the second cluster.
[0007] In one implementation, the acquiring of multiple query tasks includes: determining to acquire multiple query tasks based on the maximum concurrent number of the cluster and the running tasks of the current cluster; or acquiring multiple query tasks based on the query parameter priority of the query tasks.
[0008] In one embodiment, before determining the cluster corresponding to the query task, the method also includes: writing the query task into a database, scheduling a master node to obtain the query task from the database, the master node representing a node selected from multiple nodes deployed by the scheduler, for obtaining the query task currently required to be queried, and distributing the query task to the task execution node; based on receiving the query task at the task execution node, a query request corresponding to the query task is executed in a distributed manner.
[0009] In one embodiment, after determining the cluster corresponding to the query task based on the query task, the method further includes: in response to detecting that the second cluster is not currently within the time range of the scheduled query, adding the second cluster to the first cluster; and the query task polls the first cluster added to the second cluster.
[0010] According to a second aspect of an embodiment of the present disclosure, a distributed query device is provided, comprising a conversion unit, for, in response to obtaining a plurality of distributed query requests, converting query parameters corresponding to the query requests into query tasks based on a task model; a determination unit, for determining, based on the query task, a cluster corresponding to the query task, the cluster comprising a first cluster and a second cluster, the first cluster being used to process non-scheduled queries, the second cluster being used to process scheduled queries, determining, based on a current weight of the cluster, whether the cluster is a selected cluster or an unselected cluster, wherein the current weight of the cluster is used to determine a selected cluster for query tasks other than the first query task, determining, based on the current weight, a selected cluster corresponding to the query task, and updating the current weights of the selected cluster and the unselected cluster; a processing unit, for, based on the query task, polling the updated cluster corresponding to the query task to obtain a query result.
[0011] In one embodiment, based on the current weight, the determination unit determines the selected cluster corresponding to the query task in the following manner: in the first query, the cluster with the largest static weight is selected as the selected cluster, the query is performed, and the current weight is updated, wherein the static weight is a preset value, which is used to determine the selected cluster in the first query, and the current weight is determined based on the static weight and / or the weight change value; in the i+1th query, the current weight is updated, wherein the current weight of the selected cluster is the sum of the weight change value in the i-1th query and the static weight of the selected cluster, the current weight of the unselected cluster is the sum of the current weight in the i-1th query and the static weight of the unselected cluster, the weight change value is the difference between the current weight of the selected cluster and the weight sum value, and the weight sum value is the sum of the static weights of the selected cluster and the unselected clusters, the current weight of the cluster is judged, and the cluster with the largest current weight is selected as the selected cluster.
[0012] In one embodiment, based on the query task, the determination unit determines the cluster corresponding to the query task in the following manner: obtaining multiple query tasks and determining the query parameter identifier of the query task; if the query parameter identifier is a non-scheduled query, the query task corresponds to the first cluster; if the query parameter identifier is a scheduled query, the query task corresponds to the second cluster.
[0013] In one embodiment, the processing unit obtains multiple query tasks in the following manner: determining to obtain multiple query tasks based on the maximum concurrent number of the cluster and the running tasks of the current cluster; or obtaining multiple query tasks based on the query parameter priority of the query task.
[0014] In one embodiment, the determination unit is also used to: write the query task into a database, schedule a master node to obtain the query task from the database, the master node represents a node selected from multiple nodes deployed by the scheduler, and is used to obtain the query task currently required to be queried, and distribute the query task to the task execution node; based on the task execution node receiving the query task, the query request corresponding to the query task is executed in a distributed manner.
[0015] In one embodiment, the determining unit is further configured to: in response to detecting that the second cluster is not currently within a time range of a scheduled query, add the second cluster to the first cluster; and the query task polls the first cluster added to the second cluster.
[0016] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: execute the distributed query method described in the first aspect or any one of the implementations of the first aspect.
[0017] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the distributed query method described in the first aspect or any one of the embodiments of the first aspect.
[0018] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: in response to obtaining multiple distributed query requests, after converting the query parameters corresponding to the query requests into query tasks based on the task model, determining the selected cluster in the cluster based on the cluster corresponding to the query task and based on the current weight of the cluster, and then executing the query task based on the selected cluster to obtain the query result. Through the present disclosure, the query result is obtained by selecting the selected cluster based on the dynamic current weight, which improves the query efficiency and the flexibility of polling the query task in the cluster.
[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0021] Figure 1 The figure is a flow chart of a distributed query method according to an exemplary embodiment.
[0022] Figure 2It is a schematic diagram of a front-end template according to an exemplary embodiment.
[0023] Figure 3 It is a schematic diagram of a front-end template according to an exemplary embodiment.
[0024] Figure 4 The figure is a flow chart showing a method for determining a cluster according to an exemplary embodiment.
[0025] Figure 5 The diagram is a schematic diagram showing a method for determining a cluster according to an exemplary embodiment.
[0026] Figure 6 The figure is a flow chart of a distributed query method according to an exemplary embodiment.
[0027] Figure 7 The figure is a flowchart of a method for selecting clusters according to a dynamic rule according to an exemplary embodiment.
[0028] Figure 8 The figure is a schematic diagram showing a dynamic rule selection cluster according to an exemplary embodiment.
[0029] Fig. 9 The figure is a schematic diagram showing a dynamic rule selection cluster according to an exemplary embodiment.
[0030] Fig.10 The figure is a schematic diagram showing a dynamic rule selection cluster according to an exemplary embodiment.
[0031] Fig.11 It is a schematic diagram showing a distributed query method according to an exemplary embodiment.
[0032] Fig.12 It is a block diagram of a distributed query device according to an exemplary embodiment.
[0033] Fig.13 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0035] When performing interactive queries in related technologies, there are problems such as priority conflicts in query tasks that may delay the execution of key tasks and result in failure to obtain query results in a timely manner. Or there are problems such as unstable performance and inability to update routing strategies in real time to adapt to new situations due to changes in cluster performance over time.
[0036] In view of this, the present disclosure proposes a distributed query method that flexibly schedules different clusters, improves query efficiency, reduces user query response time and query concurrency, and thus makes query tasks more efficient when receiving multiple requests.
[0037] The distributed query method provided by the embodiment of the present disclosure is applied to the scenario of distributed ad hoc query.
[0038] Figure 1 FIG. 1 is a flow chart of a distributed query method according to an exemplary embodiment. Figure 1 As shown, the method includes steps S11 to S14.
[0039] In step S11, in response to obtaining multiple distributed query requests, query parameters corresponding to the query requests are converted into query tasks based on the task model.
[0040] In the disclosed embodiment, the query request can be understood as a request for a user to perform a query based on a specified template and query parameters of a front end, wherein the query parameters are obtained based on the query request. The task model can be understood as generating a query task related to the query parameters according to a preset task model based on the query parameters obtained from the query request when processing the query request.
[0041] For example, in order to increase the usability and efficiency of the system, the front-end page of the system can be designed as a query method based on a template. Figure 2 As shown, Figure 2 This is a schematic diagram of a front-end template according to an exemplary embodiment. Presto engine, StarRocks engine, Phoenix engine and MySQL engine can be used. You can select Presto engine to create a Structured Query Language (SQL) template, for example, it can be "selectname, dt from test where dt = '{dt}'", which can be understood as searching for names and dates in the test table and the date is {dt}, where {dt} is a parameter variable symbol, and the current parameters can be determined according to the parameters specifically entered by the user. For example Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a front-end template according to an exemplary embodiment. Figure 3As shown, the user selects the test template to be used and enters the query parameters, for example, enters "20200101" in the date query and performs the query. At this time, "selectname, dt from test where dt='{dt}'" in {dt} is "20200101", and the 20200101 string replaces the {dt} parameter variable placeholder, and the query statement finally executed is select name, dt from test where dt='20200101'. The above data is for illustrative purposes only and is not specifically limited in this disclosure.
[0042] Exemplarily, after receiving a query request from a user, query information is parsed based on the query request. As shown in Table 1, the following query information can be obtained based on the query request.
[0043]
[0044]
[0045] Table 1
[0046] Exemplarily, the task model may include multiple core fields, as shown in Table 2:
[0047]
[0048] Table 2
[0049] For example, if the current query statement is "select name from db.test limit 1", the engine used is the Presto engine, and the source is an interactive cluster (adhoc), the query parameters corresponding to the query request are converted into query tasks based on the task model, and the query tasks are the data in Table 3:
[0050]
[0051] Table 3
[0052] In step S12, based on the query task, a cluster corresponding to the query task is determined.
[0053] In the embodiment of the present disclosure, since the query task includes source, the query source corresponding to the query request can be queried, and the query scenario corresponding to the current query request can be determined. Therefore, the cluster corresponding to the query task can be determined based on the query task. Among them, the cluster includes a first cluster and a second cluster. The first cluster can be an interactive cluster (adhoc), which is used to process non-scheduled queries, can provide queries 24 hours a day, and has high resource requirements. The second cluster can be a batch cluster (batch), which is used to process scheduled queries, generally called by the system service party, and the use time period of the second cluster is fixed. For example, the query time of one cluster in the second cluster can be from 9 am to 6 pm, and the query time of another cluster can be from 5 am to 8 pm. It can be understood that when executing a query task in a cluster, the first cluster includes at least one cluster, and the second cluster includes at least one cluster. If the query task is currently selected to be executed on the first cluster, it is necessary to determine the target cluster of at least one cluster in the first cluster, and the target cluster can be called the selected cluster. The clusters other than the selected cluster in at least one cluster in the first cluster can be understood as unselected clusters. For example, the first cluster includes cluster a, cluster b, cluster c and other clusters. The first cluster is selected based on the current query task. Cluster a in the first cluster is selected, and cluster a is the selected cluster. Cluster b, cluster c and other clusters are unselected clusters.
[0054] It can be understood that the first sentence includes the selected cluster and the unselected cluster, and the second sentence includes the selected cluster and the unselected cluster.
[0055] In step S13, based on the current weight, the selected cluster corresponding to the query task is determined, and the current weights of the selected cluster and the unselected cluster are updated.
[0056] In the disclosed embodiment, the cluster corresponding to the query task is determined to be the first cluster or the second cluster based on the query task, and the selected cluster corresponding to the query task in the first cluster or the second cluster is determined based on the current weight. The current weight is used to determine the selected cluster for query tasks other than the first query task. For example, the first cluster in the current query includes cluster a, cluster b, cluster c and other clusters. The first cluster is selected based on the current query task. Based on the current weight of each cluster in the first cluster, if the current weight of cluster a is the largest, cluster a in the first cluster is selected as the selected cluster, and cluster b, cluster c and other clusters are unselected clusters.
[0057] In step S14, based on the query task, the updated cluster corresponding to the query task is polled to obtain the query result.
[0058] In the disclosed embodiment, in response to the current query task completing the query in the selected cluster, the next query task continues to query based on the cluster after the current weight is updated, so that each query task polls in different clusters to obtain query results.
[0059] In the disclosed embodiment, a user and system interaction mode based on query templates is designed to facilitate distributed and efficient query. Based on the task model, user requests are received, query tasks are generated, and query results can be returned to the caller. Further, based on the dynamic change of the current weight, the flexibility of the system is improved, the resource utilization of each cluster is improved, the load between clusters is more balanced, and the query efficiency is improved.
[0060] In the disclosed embodiment, the selected cluster corresponding to the query task is determined based on the current weight, which can be understood as a dynamic allocation method of clusters. After each update of the current weight, the corresponding current weight will be updated, and the next query task will be queried based on the updated current weight. The weights of the first cluster and the second cluster include the following: static weight, current weight, weight change value, and weight sum value.
[0061] In the embodiment of the present disclosure, at the first query, the cluster with the largest static weight is selected as the selected cluster, the query is performed, and the current weight is updated. Wherein, the static weight is a preset value, which is used to determine the selected cluster for the first query. It can be understood that in queries other than the first query, the cluster with the largest current weight is selected as the selected cluster. The current weight is determined by the static weight and / or the weight change value, the current weight of the selected cluster is determined by the static weight and the weight change value, and the current weight of the unselected cluster is determined based on the static weight. At the i+1th query, the current weight is updated, wherein the current weight of the selected cluster is the sum of the weight change value at the i-1th query and the static weight of the selected cluster, the current weight of the unselected cluster is the sum of the current weight of the i-1th query and the static weight of the unselected cluster, the weight change value is the difference between the current weight of the selected cluster and the weight sum value, and the weight sum value is the sum of the static weights of the selected cluster and the unselected cluster, the current weight of the cluster is determined, and the cluster with the largest current weight is selected as the selected cluster.
[0062] In the embodiments of the present disclosure, the calculation of the static weight, current weight, weight change value, and weight sum value is described with reference to examples, and how to select a selected cluster is described.
[0063] For example, if the current first cluster includes cluster 1, and the second cluster includes cluster 2 and cluster 3, when the cluster is constructed, since the clusters have different numbers of nodes and correspond to different resource sizes, a weight allocation strategy based on the ratio of the number of cluster nodes can be used to determine the selected cluster. It can be understood that the cluster with more nodes obtains a higher usage weight, that is, a static weight. And based on the actual performance of the cluster, the maximum number of concurrent connections of the cluster is evaluated. The details are shown in Table 4:
[0064] Cluster name Scene Identification Query time Number of nodes Static Weight Maximum number of concurrent connections Cluster 1 The first cluster All day 50 5 10 Cluster 2 Second Cluster 09:00~18:00 30 3 6 Cluster 3 Second Cluster 05:00~20:00 20 2 4
[0065] Table 4
[0066] As shown in Table 4, the static weight of cluster 1 is 5, the static weight of cluster 2 is 3, and the static weight of cluster 3 is 2. In an example, taking 6 query tasks as an example, clusters are allocated for the 6 query tasks received, as shown in Table 5:
[0067]
[0068]
[0069] Table 5
[0070] Initially, the current weights of the three clusters are 0. When the first query request is received, the allocation result is determined based on the static weight. As shown in Table 4, the static weight of cluster 1 is 5, the static weight of cluster 2 is 3, and the static weight of cluster 3 is 2. As shown in Table 5, the weight change value is the current weight change of the selected cluster after allocation, and the weight sum value is the sum of the static weight values of the clusters. Therefore, the static weight of cluster 1 is the largest at present, and cluster 1 is selected as the selected cluster to execute the query task. After executing the first allocation request, the weight values of the three clusters are changed. The change rules include the following three points: 1. Each time an allocation is requested, all clusters are traversed and their current weights are increased by the static weight of the cluster. 2. Find the cluster with the largest current weight and allocate the request to it. 3. The current weight of the selected cluster minus the total weight. Therefore, in the second request, the current weight of cluster 1 = -5 (the current weight change after the first selected cluster is assigned) + 5 (the static weight of cluster 1) = 0, the current weight of cluster 2 = 3 (the current weight of cluster 2 in the first query request) + 3 (the static weight of cluster 2) = 6, and the current weight of cluster 3 = 2 (the current weight of cluster 3 in the first query request) + 2 (the static weight of cluster 2) = 4. In the second query request, the current weight of cluster 2 is the largest, so cluster 2 is selected to query the query request this time. In the third query request, the current weight of cluster 1 = 0 (the current weight of the second query request) + 5 (the static weight of cluster 1) = 5, the current weight of cluster 2 = -4 (the current weight change of the selected cluster assigned corresponding to the second query request) + 3 (the static weight of cluster 2) = -1, and the current weight of cluster 3 = 4 (the current weight of the second query request) + 2 (the static weight of cluster 3) = 6. Therefore, in the third query request, the current weight of cluster 3 is the largest, and cluster 3 is selected to query the query request. In the fourth query request, the current weight of cluster 1 = 5 (the current weight of the third query request) + 5 (the static weight of cluster 1) = 10, the current weight of cluster 2 = -1 (the current weight of the third query request) + 3 (the static weight of cluster 2) = 2, and the current weight of cluster 3 = -4 (the current weight change of the selected cluster corresponding to the third query request) + 2 (the static weight of cluster 3) = -2. The fifth and sixth query requests are calculated as above, and the present disclosure will not explain them here.
[0071] For example, the static weight of the cluster will change, the static weight of the cluster will be modified, and the weight and value will be adjusted accordingly. The query request can continue to query the cluster after the static weight changes. Taking Table 6 as an example, Table 6 updates the static weight of cluster 3 from 2 to 6, and takes 6 queries as an example. The clusters to which each query task is assigned are shown in Table 6 below:
[0072]
[0073]
[0074] Table 6
[0075] For example, based on Table 5, the static weight 2 of cluster 3 is modified to 6. Then, in the initial value of Table 6, the current weight of cluster 1 is the current weight 0 of the 6th query request in Table 5, and the current weight of cluster 2 is the current weight change after the selected cluster corresponding to the 6th query request in Table 5 is allocated, which is -2. The current weight of cluster 3 is the current weight 2 of cluster 3 in the 6th query request in Table 5. The calculation process from the 1st query request to the 6th query request in the table is the same as the calculation process in Table 5, and the present disclosure will not discuss it here.
[0076] Exemplarily, a cluster may be removed, for example, cluster 2 may be removed. In this case, the clusters to which the six query tasks based on cluster 1 and cluster 3 are allocated are shown in Table 7 below:
[0077]
[0078] Table 7
[0079] For example, as shown in Table 7, cluster 2 is removed. Based on Table 5, the initial current weight corresponding to cluster 1 is 0, the current weight of cluster 3 is 2, and the total weight of the current cluster is 5 (static weight of cluster 1) + 2 (static weight of cluster 3) = 7. Therefore, in the first query request, the current weight of cluster 1 = 0 (the initial current weight of cluster 1) + 5 (static weight of cluster 1) = 5, and the current weight of cluster 3 = 2 (the initial current weight of cluster 1) + 2 (static weight of cluster 2) = 4. Since the current weight of cluster 1 is greater than the current weight of cluster 3, cluster 1 is selected for query this time. In the second query request, the current weight of cluster 1 = -2 (the current weight change after the selected cluster is assigned in the first query request) + 5 (static weight of cluster 1) = 3, and the current weight of cluster 3 = 4 (the current weight of cluster 3 in the first query request) + 2 (static weight of cluster 3) = 6. Since the current weight of cluster 3 is greater than the current weight of cluster 1, cluster 3 is selected for query this time. During the third to sixth query requests, the calculation process is the same as above, and the present disclosure will not discuss it here.
[0080] In the disclosed embodiment, based on the dynamically updated current weight, the selected cluster corresponding to the query task is determined, which solves the problem of maintaining cluster load balancing under the conditions of different cluster node numbers, cluster node number changes, and cluster online and offline changes, and improves the flexibility and scalability of query task polling between clusters. Through the above method, query tasks can be assigned to the appropriate cluster for execution, thereby improving the query efficiency of the query.
[0081] In the disclosed embodiment, the query parameter includes a query parameter identifier, which can be understood as a source in Table 1 and is used to determine a query scenario of the current query task.
[0082] Figure 4 FIG. 1 is a flow chart showing a method for determining a cluster according to an exemplary embodiment. Figure 4 As shown, the method includes steps S21 to S22b.
[0083] In step S21, multiple query tasks are obtained, and query parameter identifiers of the query tasks are determined.
[0084] In the disclosed embodiment, the query parameter identifier may include adhoc query (adhoc-ui) and batch query (batch). If the current query parameter identifier is adhoc query, then source=adhoc-ui in Table 1. If the current query parameter identifier is adhoc query, then source=batch in Table 1.
[0085] In step S22a, if the query parameter is identified as a non-scheduled query, the query task corresponds to the first cluster.
[0086] In step S22b, if the query parameter is identified as a scheduled query, the query task corresponds to the second cluster.
[0087] In the present disclosure, Figure 5 , Figure 5 It is a schematic diagram of a method for determining a cluster according to an exemplary embodiment. If the current source=adhoc-ui, it can be understood that the current query parameter is identified as a non-scheduled query, then the first cluster identified as adhoc is selected, that is, the interactive query cluster. If the current source=batch, it can be understood that the current query parameter is identified as a scheduled query, then the second cluster identified as batch is selected, that is, the batch processing cluster. Among them, the above determination of the corresponding cluster based on the query parameter identifier can be understood as a static rule to determine the cluster corresponding to the query task.
[0088] In the disclosed embodiment, based on static rules, the corresponding cluster can be matched through the user query parameter identifier, thereby improving the query efficiency of the query task.
[0089] In the disclosed embodiment, after receiving the query task, the number of tasks of this query task is obtained based on the scheduling strategy. Among them, the scheduling strategy can be to determine the acquisition of multiple query tasks based on the maximum concurrent number of the cluster and the running tasks of the current cluster. Or multiple query tasks are acquired based on the query parameter priority of the query task. If the query task identifies the query parameter priority, the query tasks are sequentially taken out from the task pool in the order of the query parameter priority. The query tasks with higher priorities are executed first. If the user does not specify the priority of the query task, the default priority is 0. If the query parameter priorities are the same, the creation time of the query tasks is compared, and the query tasks submitted first are executed first.
[0090] In the disclosed embodiment, multiple query tasks are obtained based on the scheduling test, which can support a higher user query concurrency and improve the query efficiency of the query tasks.
[0091] In the disclosed embodiment, before determining the cluster corresponding to the query task, it is also necessary to deploy nodes based on the scheduler and the task execution module to distribute the query task.
[0092] Figure 6 FIG. 1 is a flow chart of a distributed query method according to an exemplary embodiment. Figure 6 As shown, the method includes steps S31 to S32.
[0093] In step S31, the query task is written into the database, and the scheduling master node obtains the query task from the database.
[0094] In the disclosed embodiment, the master node represents a node selected from multiple nodes deployed by the scheduler, which is used to obtain the query task that needs to be queried currently and distribute the query task to the task execution node. It is understandable that only the selected node among multiple nodes can serve as the master node, and the unselected nodes are responsible for monitoring the survival status of the master node. If the current master node fails, a re-election can be performed in other nodes to ensure the normal operation and availability of the scheduler.
[0095] In the disclosed embodiment, the nodes successfully registered by the task execution module, the access addresses (Internet Protocol, IP) and ports (port) of the nodes are obtained through the registered network to form a node list, and based on the scheduling strategy, the query tasks whose current task status is "to be executed" are queried in the database, and the "to be executed" query tasks are allocated to the node list based on the dynamic allocation method of the cluster.
[0096] In step S32, based on the task execution node receiving the query task, the query request corresponding to the query task is executed in a distributed manner.
[0097] In the disclosed embodiment, the task execution module can deploy multiple nodes. After the node is started, it first registers and writes the IP and port. A network interface can be provided in the task execution module, and the request path is receive / {sid} / , which is used to receive the tasks distributed by the master node, and select a cluster based on the dynamic current weight of the cluster, and execute the query request corresponding to the query task in a distributed manner.
[0098] In the disclosed embodiment, after the current query task is completed, the query result is stored in the database, and at the same time, the task status of the corresponding query task in the database is updated to "execution completed".
[0099] In the disclosed embodiment, multiple nodes are deployed to have high availability, and the query task is assigned to the task execution module through the master node to reduce the waiting response time of the user query. Multiple nodes are deployed in the task execution module to be responsible for distributed execution of the query task assigned by the master node, thereby improving the query efficiency.
[0100] In the embodiment of the present disclosure, after a cluster is selected based on static rules, a suitable cluster is further dynamically selected based on dynamic rules. Figure 7 FIG. 1 is a flow chart showing a method for selecting clusters according to a dynamic rule according to an exemplary embodiment. Figure 7 As shown, the method includes steps S41 to S42.
[0101] In step S41 , in response to detecting that the second cluster is not currently within the time range of the scheduled query, the second cluster is added to the first cluster.
[0102] In the embodiment of the present disclosure, in response to the second cluster not being within the time range of the scheduled query, the second cluster is currently in an idle state, and the second cluster can be added to the row of the first cluster as the row of the current query task selection cluster.
[0103] In step S42, the query task polls the first cluster that has joined the second cluster.
[0104] In the present disclosure, Figure 8 , Figure 8 FIG. 1 is a schematic diagram showing a dynamic rule selection cluster according to an exemplary embodiment. Figure 8 It can be seen that based on the static rules, the clusters where query tasks can be executed can be identified. The master node will allocate query tasks with adhoc identifier to the first cluster for query based on the dynamic allocation method of the cluster, and allocate query tasks with batch identifier to the second cluster for query. Figure 8Cluster 1 (cluster1) is the first cluster, which can be queried 24 hours a day, cluster 2 (cluster2) is a cluster in the second cluster, and the queryable time period is from 9 am to 6 pm, and cluster 3 (cluster3) is a cluster in the second cluster, and the queryable time period is from 5 am to 8 pm. If the current time is 20:00, that is, 8 pm, cluster1, cluster2 and cluster3 are all in the idle time period, and cluster 2 and cluster 3 can be added to adhoc. It can be understood that the available clusters of adhoc cluster type are currently cluster 1, cluster 2 and cluster 3. There are currently no available clusters for batch cluster type. In the adhoc cluster type, the above three clusters can be selected for query by the query task based on the current weight of the cluster itself.
[0105] In the following examples of this disclosure and in the accompanying drawings, cluster1 refers to cluster 1, cluster2 refers to cluster 2, and cluster3 refers to cluster 3.
[0106] For example, Fig. 9 As shown, Fig. 9 is a schematic diagram of a dynamic rule selection cluster according to an exemplary embodiment. If the current time is 18:00, that is, 6:00 p.m., cluster1 and cluster2 are currently in an idle time period, and one of the clusters can be selected for query based on the dynamic allocation method of the cluster. Specifically, Fig. 9 It can be understood that the current time is 18:00, and cluster2 is not in the available period of batch. Add cluster2 to the list of available clusters of adhoc. It can be understood that when the current time is 18:00, the available clusters of adhoc cluster type include cluster 1 and cluster 2, and the available clusters of batch cluster type include cluster 3. Update the dynamic allocation algorithm of adhoc cluster, add cluster2 and the current weight. Remove cluster2 from the list of available clusters of batch, update the dynamic allocation algorithm of batch group, remove cluster2, so that the subsequent batch tasks will not poll cluster2.
[0107] For example, Fig.10 FIG. 1 is a schematic diagram showing a dynamic rule selection cluster according to an exemplary embodiment. Fig.10As shown in the figure, if the current time is updated from 18:00 to 20:00, cluster3 is not in the batch available time period, and cluster3 is added to the adhoc available cluster list. At this time, the available clusters of the adhoc cluster type include cluster 1, cluster 2, and cluster 3. Update the dynamic allocation algorithm of the adhoc cluster, add cluster3 and dynamic weight. And remove cluster3 from the batch available cluster list, so that the subsequent batch tasks will not poll cluster3.
[0108] In the disclosed embodiment, dynamic rules match cluster labels and analyze the available time of the clusters to obtain a list of queryable clusters. The cluster-based dynamic allocation algorithm can intelligently allocate query requests to suitable clusters for execution, thereby improving query efficiency and improving the system's throughput during business peak periods, making the distributed query method more efficient.
[0109] The present disclosure embodiment combines Fig.11 The distributed query method is described. Fig.11 It is a schematic diagram showing a distributed query method according to an exemplary embodiment.
[0110] In the embodiment of the present disclosure, the modules used in the distributed query method include a front-end module, an interaction module, a scheduler module and an execution module.
[0111] In the disclosed embodiment, in the front-end module, an SQL template can be created, and the user specifies the template and parameter variables to obtain a query request for query. The query request is sent to the interactive module, and after the query result is obtained, the query result can be visualized through the front-end module.
[0112] In the embodiment of the present disclosure, in the interactive module, the query parameters corresponding to the query request are converted into query tasks based on the task model. The task model is persisted in the database, and an interface for task query is provided to the user so that the unique identifier of the query task, the task status, etc. are returned to the user. In the interactive module, a key-value (Key-Value, KV) database can also be connected to provide the user with an interface for obtaining query results. If the task obtained by the user through the task status query interface is "executed successfully", the data query results can be returned to the user through the interface. The implementation logic of the query result interface is to query the SQL execution result from the KV database by using the unique identifier sid of the task as the key in the key-value pair.
[0113] In the disclosed embodiment, the scheduler module includes multiple schedulers, and multiple nodes can be deployed. After each scheduler is running, a master node election is to be carried out, wherein the election is carried out through the Apache Curator framework (a Java client library and framework for simplifying the use of Apache ZooKeeper). The scheduler that successfully elects the master has the master node and performs task distribution based on the scheduler scheduling policy. The scheduler that fails to elect the master is responsible for monitoring the survival status of the master scheduler, and re-election after the master scheduler fails, thereby ensuring the availability of the system.
[0114] In the disclosed embodiment, the execution module performs node registration, and the task execution module can deploy multiple nodes. It can register with zookeeper through Apache Curator. After receiving the task distributed by the scheduler module, the engine is selected for execution according to the engine information in the SQL task. If there are multiple clusters of engines, the cluster is selected through the multi-cluster intelligent selection strategy, that is, the multi-cluster routing strategy. First, find the list of queryable clusters based on static rules, and execute SQL on the cluster corresponding to the request load based on dynamic rules. Exemplarily, you can choose the StarRocks query engine, select cluster 1 or cluster 2 or other clusters, you can also choose the presto query engine, select cluster 1 or cluster 2 or other clusters, or you can choose other query engines, select cluster 1 or cluster 2 or other clusters. After querying based on the query engine, the data is stored in the KV database, the key is sid, and the value is the query result. At the same time, the task status in the database is updated to "execution completed".
[0115] In the disclosed embodiment, in the ad hoc query scenario, through the task model, through the collaborative work of the interaction module, the scheduler module and the execution module, an efficient query based on distribution is realized. The interaction module can be responsible for receiving user requests, generating query tasks, and returning query results to the calling segment. The scheduler module can deploy multiple nodes, and through zookeeper to elect, it has high availability, and it is responsible for assigning query tasks to the task execution module. The task execution module can deploy multiple nodes, which is responsible for distributed execution of query tasks assigned by the scheduler module. The front-end module provides an interactive mode based on SQL templates, which greatly improves the user SQL development efficiency and system usability. Through the designed and implemented distributed query system, the waiting response time of user queries is greatly reduced, and the query efficiency is improved. It can support higher user query concurrency, and has distributed scalability and fault tolerance capabilities, which improves the availability and stability of the system using the distributed query method. A dynamic allocation algorithm for a cluster is implemented, and based on the current weight of each cluster, requests are distributed to the cluster in a smoother polling manner, so that the load of each cluster is more balanced. Based on the algorithm, dynamic changes of clusters and weights are supported to improve the flexibility of the system. Through static rules and dynamic rules, the multi-cluster intelligent routing strategy for query tasks is implemented. Static rules match the corresponding cluster labels through user query features, and dynamic rules match cluster labels and analyze the available time of the cluster to obtain a list of queryable clusters. Based on the multi-cluster dynamic smooth weighted polling algorithm, SQL is intelligently assigned to the most suitable cluster for execution. Through the above optimization measures, the resource utilization of each cluster is improved, the load between clusters is more balanced, and the query efficiency is further improved.
[0116] Based on the same concept, the embodiment of the present disclosure also provides a distributed query device 100.
[0117] It is understandable that the distributed query device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0118] Fig.12 1 is a block diagram of a distributed query device 100 according to an exemplary embodiment. Fig.12 The device includes a conversion unit 101, a determination unit 102 and a processing unit 103.
[0119] The conversion unit 101 is configured to, in response to obtaining a plurality of distributed query requests, convert query parameters corresponding to the query requests into query tasks based on a task model.
[0120] A determination unit 102 is used to determine a cluster corresponding to a query task based on the query task, where the cluster includes a first cluster and a second cluster, where the first cluster is used to process non-scheduled queries and the second cluster is used to process scheduled queries, and where the cluster is determined to be a selected cluster or an unselected cluster based on a current weight of the cluster, where the current weight of the cluster is used to determine a selected cluster for query tasks other than the first query task, and where the selected cluster corresponding to the query task is determined based on the current weight, and the current weights of the selected cluster and the unselected cluster are updated.
[0121] The processing unit 103 polls the updated cluster corresponding to the query task based on the query task to obtain the query result.
[0122] In one embodiment, based on the current weight, the determination unit 102 determines the selected cluster corresponding to the query task in the following manner: in the first query, the cluster with the largest static weight is selected as the selected cluster, the query is performed, and the current weight is updated, wherein the static weight is a preset value, which is used to determine the selected cluster in the first query, and the current weight is determined based on the static weight and / or the weight change value; in the i+1th query, the current weight is updated, wherein the current weight of the selected cluster is the sum of the weight change value in the i-1th query and the static weight of the selected cluster, the current weight of the unselected cluster is the sum of the current weight in the i-1th query and the static weight of the unselected cluster, the weight change value is the difference between the current weight of the selected cluster and the weight sum value, and the weight sum value is the sum of the static weights of the selected cluster and the unselected clusters, the current weight of the cluster is judged, and the cluster with the largest current weight is selected as the selected cluster.
[0123] In one implementation, based on the query task, the determination unit 102 determines the cluster corresponding to the query task in the following manner: obtain multiple query tasks and determine the query parameter identifier of the query task; if the query parameter identifier is a non-scheduled query, the query task corresponds to the first cluster; if the query parameter identifier is a scheduled query, the query task corresponds to the second cluster.
[0124] In one implementation, the processing unit 103 acquires multiple query tasks in the following manner: acquiring multiple query tasks based on the maximum concurrent number of the cluster and the running tasks of the current cluster; or acquiring multiple query tasks based on the query parameter priority of the query task.
[0125] In one embodiment, the determination unit 102 is also used to: write the query task into the database, schedule the master node to obtain the query task in the database, the master node represents a node selected from multiple nodes deployed by the scheduler, and is used to obtain the query task that currently needs to be queried, and distribute the query task to the task execution node; based on the task execution node receiving the query task, the query request corresponding to the query task is executed in a distributed manner.
[0126] In one implementation, the determining unit 102 is further configured to: in response to detecting that the second cluster is not currently within the time range of the scheduled query, add the second cluster to the first cluster; and perform query tasks in the first cluster added to the second cluster.
[0127] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0128] like Fig.13 As shown, Fig.13 : is a schematic diagram of an electronic device provided by an embodiment of the present invention. One embodiment of the present invention provides an electronic device 200. Among them, the electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. Among them, the memory 201 is used to store instructions. The processor 202 is used to call the instructions stored in the memory 201 to execute the file merging method of the embodiment of the present invention. Among them, the processor 202 is connected to the memory 201 and the I / O interface 203 respectively, for example, it can be connected through a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program of the file merging method involved in the embodiment of the present invention, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the program stored in the memory 201.
[0129] In the embodiment of the present invention, the processor 202 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 202 can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.
[0130] The memory 201 in the embodiment of the present invention may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD).
[0131] In the embodiment of the present invention, the I / O interface 203 may be used to receive input instructions (such as digital or character information, and key signal input related to user settings and function control of the electronic device 200), and may also output various information to the outside (such as images or sounds). In the embodiment of the present invention, the I / O interface 203 may include one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.
[0132] It is to be understood that, although the operations are described in a specific order in the drawings in the embodiments of the present invention, it should not be understood as requiring to perform these operations in the specific order shown or in a serial order, or requiring to perform all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0133] The methods and devices involved in the embodiments of the present invention can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "device" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.
[0134] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.
[0135] The foregoing description of the implementation of the present invention has been given for purposes of illustration and description. The foregoing description is not intended to be exhaustive nor to limit the invention to the exact form disclosed, and various variations and modifications are possible in accordance with the above teachings or may be obtained from the practice of the invention. These embodiments are selected and described in order to illustrate the principles of the invention and its practical application, so that those skilled in the art can utilize the invention in various embodiments and with various modifications suitable for the particular use contemplated.
Claims
1. A distributed query method, characterized in that: include: In response to obtaining a plurality of distributed query requests, converting query parameters corresponding to the query requests into query tasks based on the task model; Based on the query task, determine a cluster corresponding to the query task, the cluster includes a first cluster and a second cluster, the first cluster is used to process non-scheduled queries, and the second cluster is used to process scheduled queries, and determine whether the cluster is a selected cluster or an unselected cluster based on a current weight of the cluster, wherein the current weight of the cluster is used to determine the selected cluster for query tasks other than the first query task; Based on the current weight, determining a selected cluster corresponding to the query task, and updating the current weights of the selected cluster and the unselected cluster; Based on the query task, the updated cluster corresponding to the query task is polled to obtain a query result.
2. The method according to claim 1, characterized in that The determining, based on the current weight, a selected cluster corresponding to the query task includes: In the first query, the cluster with the largest static weight is selected as the selected cluster, the query is performed, and the current weight is updated, wherein the static weight is a preset value used to determine the selected cluster in the first query, and the current weight is determined based on the static weight and / or the weight change value; At the i+1th query, the current weight is updated, wherein the current weight of the selected cluster is the sum of the weight change value at the i-1th query and the static weight of the selected cluster, the current weight of the unselected cluster is the sum of the current weight at the i-1th query and the static weight of the unselected cluster, the weight change value is the difference between the current weight of the selected cluster and the weight sum value, the weight sum value is the sum of the static weights of the selected cluster and the unselected clusters, the current weight of the cluster is determined, and the cluster with the largest current weight is selected as the selected cluster.
3. The method according to claim 1, characterized in that The determining, based on the query task, a cluster corresponding to the query task includes: Acquire multiple query tasks, and determine query parameter identifiers of the query tasks; If the query parameter indicates a non-scheduled query, the query task corresponds to the first cluster; If the query parameter identifier is a scheduled query, the query task corresponds to the second cluster.
4. The method according to claim 3, characterized in that The obtaining of multiple query tasks includes: Determine to obtain multiple query tasks based on the maximum concurrent number of the cluster and the running tasks of the current cluster; or A plurality of query tasks are acquired based on the query parameter priorities of the query tasks.
5. The method according to claim 1, characterized in that: Before determining the cluster corresponding to the query task, the method further includes: The query task is written into a database, and a scheduling master node obtains the query task from the database. The master node represents a node selected from multiple nodes deployed by the scheduler, and is used to obtain the query task that needs to be queried currently, and distribute the query task to the task execution node; The query task is received based on the task execution node, and the query request corresponding to the query task is executed in a distributed manner.
6. The method according to claim 3, characterized in that: After determining the cluster corresponding to the query task based on the query task, the method further includes: In response to detecting that the second cluster is not currently within a time range of the scheduled query, adding the second cluster to the first cluster; The query task polls the first cluster that joins the second cluster.
7. A distributed query device, characterized in that: include: A conversion unit, configured to, in response to obtaining a plurality of distributed query requests, convert query parameters corresponding to the query requests into query tasks based on a task model; A determination unit, configured to determine, based on the query task, a cluster corresponding to the query task, the cluster comprising a first cluster and a second cluster, the first cluster being used to process non-scheduled queries, and the second cluster being used to process scheduled queries, and to determine, based on a current weight of the cluster, whether the cluster is a selected cluster or an unselected cluster, wherein the current weight of the cluster is used to determine a selected cluster for query tasks other than the first query task, and to determine a selected cluster corresponding to the query task based on the current weight, and to update the current weights of the selected cluster and the unselected cluster; The processing unit, based on the query task, polls the updated cluster corresponding to the query task to obtain a query result.
8. The device according to claim 7, characterized in that Based on the current weight, the determination unit determines the selected cluster corresponding to the query task in the following manner: In the first query, the cluster with the largest static weight is selected as the selected cluster, the query is performed, and the current weight is updated, wherein the static weight is a preset value used to determine the selected cluster in the first query, and the current weight is determined based on the static weight and / or the weight change value; At the i+1th query, the current weight is updated, wherein the current weight of the selected cluster is the sum of the weight change value at the i-1th query and the static weight of the selected cluster, the current weight of the unselected cluster is the sum of the current weight at the i-1th query and the static weight of the unselected cluster, the weight change value is the difference between the current weight of the selected cluster and the weight sum value, the weight sum value is the sum of the static weights of the selected cluster and the unselected clusters, the current weight of the cluster is determined, and the cluster with the largest current weight is selected as the selected cluster.
9. An electronic device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the distributed query method according to any one of claims 1 to 6.
10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the distributed query method according to any one of claims 1 to 6.
