A big data heterogeneous query scheduling method based on cache multiplexing

Abstract

The application discloses a big data heterogeneous query scheduling method based on cache multiplexing, and comprises the following steps: obtaining information of a user's submitted query; submitting all queries to a to-be-executed queue; classifying the queries according to big data query types and respectively calling the following steps; determining a historical query window size according to a cache system size; calculating query cache proportions of all queries in the to-be-executed queue according to data tables accessed by the queries in the historical query window; placing a query with the highest query cache proportion into the to-be-executed queue; and submitting the queries in the to-be-executed queue to a big data query system in sequence. In the big data query scheduling scene, the application fully considers query types and cache states to perform scheduling, reduces cache replacement frequency and improves overall execution efficiency of a query sequence. The application solves the problem of low performance caused by the difficulty of a big data query system in perceiving a data cache state.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cloud computing system scheduling management, and particularly relates to a heterogeneous query scheduling method based on cache multiplexing. BACKGROUND

[0002] A big data query system is a computing system for querying and analyzing large amounts of data. It can help users quickly retrieve and analyze massive amounts of data to better understand the data and make better decisions, and can handle semi-structured and unstructured data, making up for the shortcomings of traditional data analysis techniques. Common big data query systems include Apache Spark, Apache Hive, Presto, etc.

[0003] However, hot spot phenomena of data access are very common in big data queries. In corporate business data analysis and ad hoc queries, some data tables or some columns of data tables are frequently accessed. When a data table is read, the data table is cached, and subsequent queries can read from the cache to speed up execution as long as the data table is not evicted. However, in a high-concurrency environment and a cloud-native multi-tenant environment, many different data tables are accessed at the same time, which increases the cache replacement frequency and reduces the probability of query acceleration through the cache.

[0004] However, the existing multi-job scheduling strategy has some problems in big data query systems. First, big data queries involve reading multiple data tables, making the cached data tables in the cache system uncertain during execution, so it is not possible to generate scheduling results for all queries at the query submission stage. This real-time computing approach increases query latency. Second, new data tables are constantly introduced during query execution, causing cache replacement and uncertainty in the cache state, which makes it impossible to calculate a high-performance scheduling result at the query submission stage. In addition, due to the existence of different types of queries (such as interactive analysis and ETL jobs) in big data query systems, if all queries are treated equally during scheduling, some queries with higher latency requirements may have longer waiting times. SUMMARY

[0005] The present application relates to the field of cloud computing system scheduling management, and particularly relates to a heterogeneous query scheduling method based on cache multiplexing.

[0006] Technical solution: To achieve the above-mentioned application purpose, the technical solution adopted by the present application is a heterogeneous query scheduling method based on cache multiplexing, comprising the following steps:

[0007] (1) The user submits a big data query description to the scheduling controller. The query description includes information such as the query type, query statement, query operating environment, and data tables required for the query;

[0008] (2) The scheduling controller senses the big data query submitted in (1) and adds the relevant query description to the initial submission queue in the scheduling controller; the initial submission queue contains multiple queries initially submitted by users;

[0009] (3) classifying all initially submitted queries according to the query type in the query description in the initially submitted queue, and executing a scheduling method for queries of the same category;

[0010] (4) Obtaining the cache system storage capacity based on the query description and determining the historical query window size based on the size of the query access data table recently submitted to the pending execution queue; the pending execution queue is the queue to which the selected queries are submitted after the scheduling algorithm. The historical query window is a query set that best reflects the cache system data storage status during the current execution process;

[0011] (5) Obtain all query descriptions in the historical query window obtained in step (4), and obtain a data table description that needs to be accessed by the query based on each query description;

[0012] (6) Based on the data table description obtained in step (5), the data table cache ratio of each query is calculated for all queries in the initial submission queue, and the query with the highest data table cache ratio is selected and submitted to the queue to be executed; the data table cache ratio represents the ratio of the data required to access the query in the cache system, that is, the ratio of all data in the historical query window.

[0013] (7) Submit queries to the big data query system in sequence according to the query description order of the queue to be executed in step (6). The big data query system is responsible for the subsequent query execution process.

[0014] Furthermore, in step (4), the size of the historical query window can be changed dynamically. Each time a new query is added to the queue to be executed, the size of the data table it accesses is constantly changing. In order to make the scheduling result more accurate, the historical query window will also change accordingly. There is an initial submission queue, a queue to be executed after scheduling, and a cache system size. The historical query window size calculation process will traverse the query from the end of the queue to be executed to determine whether the sum of the data tables used by the query and the data tables required for the query in the historical query window exceeds the cache system size. If it does not exceed the cache system size, the query will be added to the historical query window and the traversal will continue. Otherwise, the calculation is completed.

[0015] Furthermore, in step (5), a historical query window is used to replace the cache system data cache status; in a big data query system, big data queries often involve reading multiple data tables, which makes the data tables cached in the cache system in an uncertain state during the execution process, and it is impossible to generate the scheduling results of all queries in time; although the cache status during the query execution process cannot be obtained in the query submission stage, the data tables read by the query can be analyzed through the historical query window. Since the data tables read by the query will be cached, the probability of these data tables being in the cache is high, so the scheduling results of all queries can be generated in time.

[0016] Furthermore, in step (6), a mathematical modeling method is used to quantitatively calculate the priority score of each query; the mathematical modeling method is based on the data table in the historical query window, calculates the table cache ratio of all queries in the initial submission queue and uses it as the priority for scheduling; let the data table set recentTables read by all queries in the historical query window be {t1, t2…, t n}, t k (1≤k≤n) represents the data table of the kth historical query, and the size of each table is s1, s2…, s n , s k (1≤k≤n) represents t k The size of the table, the set of all pending queries q init For {q1,q2…,q n},q k (1≤k≤n) represents the kth query to be executed, where the set of data tables required for the i-th (1≤i≤n) query is {t i1 ,t i2 …,t ij}, t ik (1≤k≤j) represents one of the data tables, and the size of each table is s i1 ,s i2 …,s ij , s ik (1≤k≤j) represents t ik The size of the data table, the size of the data table in the cache system for the i-th query s cached_table =∑ k ∏(t ik inrecentTables)*s ik , t ik and s ik See the above definition, Π represents selection: select the data table size s that appears in recentTables ik , the data table size s in the cache system capacity_table =∑ ks k , s k See the above definition, the priority cacheRatio of the i-th query i cacheRatio i =s cached_table ÷s capacity_table ; Each time scheduling is performed, the priority of all queries in the initial submission queue is calculated, and the query with the highest priority is selected and placed in the queue to be executed.

[0017] Beneficial effects of the present invention: The scheduling technology method described in the present invention can adjust the scheduling priority of the query according to the query type and the cache system status, improve the utilization rate of the cache system, and improve the efficiency of query sequence execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0019] Figure 2 This is a comparison chart of the experimental results of the present invention on the running time of the initial submission sequence on the artificially generated submission sequences containing 20, 35, and 50 respectively;

[0020] Figure 3 This is an experimental comparison chart of the present invention performing interactive analysis query waiting time on 20, 35, and 50 artificially generated submission sequences, compared with the initial submission sequence. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0022] The present invention proposes a large data heterogeneous query scheduling method based on cache reuse, which solves the problem of difficulty in reusing cache of data tables in multi-tenant sharing in existing methods. Figure 1 As shown, the complete process of the present invention includes 7 steps. The following takes 6 different queries (the queries are marked as QA, QB, QC, QD, QE, and QF, among which interactive analysis has three queries QA, QB, and QC, and ETL jobs have three queries QD, QE, and QF. QA and QD use data tables TA and TB, QB and QE use data tables TA and TC, and QC and QF use data table TA. The queue to be executed has QG, QH, and QI queries, which use data tables TD, TE, and TB respectively. All the above data tables are of the same size) as an example to illustrate the specific implementation of the present invention.

[0023] The specific implementation of the step (1) of the technical solution is that a user fills in the related configuration parameters of the big data query description (i.e., the six queries of QA, QB, QC, QD, QE and QF) according to the own needs, and submits to the cluster. The configuration parameters include a query statement, a query type, a running cluster of query submission and data table information required for the query. The data table information must be pre-stored in the cache system started in the cluster. The above query description is submitted to the central coordination database (such as etcd) in the cluster.

[0024] The specific implementation of the step (2) of the technical solution is that the scheduling controller designed by the application is pre-registered to the central coordination database (etcd), receives a new big data query submission event through an event-driven mechanism, and pulls the query description information from the central coordination database and adds to the initial submission queue.

[0025] The specific implementation of the step (3) of the technical solution is that the scheduling controller designed by the application traverses the initial submission queue, classifies the queries and then schedules respectively, i.e., executes the scheduling algorithm for QA, QB, QC and QD, QE and QF respectively.

[0026] The specific implementation of the step (4) of the technical solution is that the scheduling controller designed by the application first acquires the size of the cache system (assuming that the cache system can store two TA data tables), and then traverses the query description from the tail of the to-be-executed queue to determine the historical query window. In this example, QH and QI are in the historical query window, and when QG is calculated, QG, QH and QI access data tables exceeding the cache system size, so they are not in the historical query window.

[0027] The specific implementation of the step (5) of the technical solution is that the scheduling controller designed by the application acquires the query description QH and QI in the historical query window, and acquires the data tables TE and TB accessed by QH and QI.

[0028] The specific implementation of the step (6) of the technical solution is that the scheduling controller designed by the application calculates the priority of the interactive analysis queries QA, QB and QC in the initial submission queue according to the data tables TE and TB acquired in the step (5). According to the scheduling algorithm of the application, the data table set recentTables read by all queries in the historical query window is {TE, TB}, the size of each table is s E ,s B , the to-be-executed query set q init is {QA, QB, QC}, wherein the data table set required by the QA query is {TA, TB}, the size of each table is s A ,s B , and the data table size scachedtable =∑ k ∏(t ik in recentTables)*s ik =s B , the size of data table in cache system s capacitytable =∑ k s k =s E +s B , the priority of QA query cacheRatio i is cacheRatio i =s cached_table ÷s capacity_table =s B ÷(s E +s B ). Other same reason, the calculation of QA has higher priority, so it will be submitted to the execution queue, the step will be repeated until the interactive analysis query is completed, after the interactive analysis query scheduling is completed, the ETL job (QD, QE, QF) is scheduled. The step will be repeated until all queries are completed.

[0029] The specific implementation of step (7) of the technical solution is that the scheduling controller designed by the application submits the queries in the execution queue to the big data query system in sequence for execution.

[0030] To test the actual performance of the method, TPC-DS, TPC-H, SSB three data sets and the execution sequences with lengths of 20, 35 and 50 composed of three corresponding query examples (containing 25% of interactive analysis queries and 75% of ETL jobs) are used for benchmark testing. The test compares the execution time of the initial submission sequence and the execution sequence optimized by the scheduling algorithm and the waiting time of the interactive analysis query. The test results are shown in Table 1 and Table 2. Figure 2 、 Figure 3 The big data heterogeneous query scheduling method based on cache reuse proposed by the application has shorter execution time and waiting time of interactive analysis query than the initial submission sequence, which shows that the method can effectively improve the scheduling performance and effectively utilize the cache resources.

[0031] In summary, in the big data query scheduling scenario, the application fully considers the query type and cache state for scheduling, reduces the cache replacement frequency and improves the overall execution efficiency of the query sequence, and solves the problem of low performance caused by the difficulty of the big data query system to perceive the data cache state.

Claims

1. A large data heterogeneous query scheduling method based on cache reuse, comprising the following steps: (1) A user submits a large data query description to a scheduling controller, and the query description contains query type, query statement, query execution environment, and query required data table information; (2) The scheduling controller perceives the large data query submitted in step (1), and adds the relevant query description to an initial submission queue in the scheduling controller; the initial submission queue contains multiple queries initially submitted by users; (3) All initially submitted queries are classified according to the query type in the query description in the initial submission queue, and the scheduling method described in steps (4) to (7) is performed on queries of the same category; (4) The cache system storage capacity is obtained according to the query description, and the historical query window size is determined according to the size of the data table accessed by the query recently submitted to the execution queue; the execution queue is a queue in which selected queries are submitted after the scheduling algorithm, and the historical query window is a query set that can reflect the data storage state of the cache system in the current execution process as much as possible; (5) All query descriptions in the historical query window obtained in step (4) are obtained, and the data table description required by each query description is obtained according to each query description; (6) According to the data table description obtained in step (5), the data table cache ratio of each query in the initial submission queue is calculated, and the query with the highest data table cache ratio is selected and submitted to the execution queue; the data table cache ratio represents the proportion of the data accessed by the query in the cache system; (7) The query description sequence of the execution queue in step (6) is sequentially submitted to the large data query system, and the large data query system is responsible for the subsequent query execution process. 2.The method of claim 1, wherein: In step (4), the historical query window size can dynamically change; each time a new query is added to the execution queue, the historical query window also changes; there is an initial submission queue, a scheduled execution queue, and a cache system size; the historical query window size calculation process traverses the queries from the tail of the execution queue to the front, judges whether the sum of the data tables used by the query and the data tables required by the queries in the historical query window exceeds the cache system size, if not, the query is added to the historical query window and the traversal continues, otherwise the calculation is completed. 3.The method of claim 1, wherein: In step (5), the historical query window is used to replace the cache system data cache state.

4. The method of claim 1, wherein the method is based on cache reuse for heterogeneous query scheduling of big data. In step (6), a mathematical modeling method is used to quantitatively calculate the priority score of each query; the mathematical modeling method is based on the data table in the historical query window, calculates the table cache ratio of all queries in the initial submission queue and uses it as the priority for scheduling; let the data table set recentTables read by all queries in the historical query window be {t1, t2…, t n }, t k Represents the data table of the kth historical query, 1≤k≤n, and the size of each table is s1,s2…,s n , s k Indicates t k The size of the table, 1≤k≤n, the set of all pending queries q init For {q1,q2…,q n },q k Indicates the kth query to be executed, 1≤k≤n, where the set of data tables required for the i-th query is {t i1 ,t i2 …,t ij }, 1≤i≤n, t ik Represents one of the data tables, 1≤k≤j, and the size of each table is s i1 ,s i2 …,s ij , s ik Indicates t ik The size of the data table, 1≤k≤j, the size of the data table in the cache system for the i-th query s cached_table =∑ k ∏(t ik in recentTables)*s ik , Π means selection: select the data table size s that appears in recentTables ik , the data table size s in the cache system capacity_table =∑ k s k , the priority cacheRatio of the i-th query i cacheRatio i =s cached_table ÷s capacity_table ; Each time scheduling is performed, the priority of all queries in the initial submission queue is calculated, and the query with the highest priority is selected and placed in the queue to be executed.

Images