Data query method and related device

Abstract

Disclosed in embodiments of the present application is a data query method. In the method, after a data query task is split into a plurality of sub-tasks, a feasible processing method for each sub-task can be determined, and then a plurality of task combinations are obtained on the basis of the feasible processing method for each sub-task. The processing method for each sub-task in a task combination will affect the query cost during execution of the data query task based on the task combination, that is, under different combinations of the plurality of sub-tasks and the corresponding processing methods, the comprehensive cost, i.e., the execution efficiency and the execution cost, of the data query task usually varies. On this basis, in the method, a target task combination having the lowest query cost can be determined from a plurality of task combinations by means of a cost estimation model, thereby adaptively implementing allocation of stream and batch tasks, making full use of resources of a batch and stream integrated data processing framework, and implementing data query that achieves both optimal query efficiency and optimal query cost to obtain a query result.

Description

A data query method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 19, 2023, with Chinese application number 202310889683.X and invention name “A task processing method, device and computing device cluster”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 29, 2023, with Chinese application number 202311863809.2 and invention name “A data query method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data query technology, and in particular to a data query method and related equipment. Background Art

[0003] As business scenarios continue to expand, practical applications often face the need to process both stream and batch data. To address this, data processing frameworks such as Spark and Flink currently support integrated batch and stream processing.

[0004] However, in actual use, users need to manually allocate stream processing tasks and batch processing tasks required to complete data queries, so that the data processing framework can call stream processing resources to implement manually allocated stream processing tasks, and call batch processing resources to implement manually allocated batch processing tasks. This processing flow relies on manual experience to implement task allocation, which may not make good use of the resources of the data processing framework, resulting in the execution method of data query tasks not being the best method.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide a data query method to address the problem of relying on manual experience to allocate stream processing tasks and batch processing tasks, which may not effectively utilize the resources of the data processing framework and result in a suboptimal execution method for data query tasks. The present application also provides corresponding apparatus, devices, computer-readable storage media, and computer program products.

[0007] The first aspect of the present application provides a data query method, which includes: obtaining a query statement, which is used to issue a data query task; determining multiple task combinations, wherein each task combination includes multiple subtasks, and the multiple subtasks are obtained by splitting the data query task, and the processing method of each subtask is batch processing or stream processing, wherein the multiple task combinations include a first task combination and a second task combination, and the first task combination and the second task combination include at least one different subtask, or the multiple task combinations include a third task combination and a fourth task combination, and the multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing methods of the multiple subtasks in the third task combination and the combination of the processing methods of the multiple subtasks in the fourth task combination are different; based on a cost evaluation model, determining a target task combination from the multiple task combinations, and the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations; performing data query according to the target task combination to obtain a query result corresponding to the data query task.

[0008] In the first aspect, after splitting a data query task into multiple subtasks, feasible processing methods for each subtask can be determined, thereby obtaining multiple task combinations based on the feasible processing methods for each subtask. The processing methods of the subtasks in a task combination affect the query cost when executing the data query task based on that task combination. In other words, the combined execution efficiency and execution cost of the data query task often differ under different combinations of multiple subtasks and corresponding processing methods.

[0009] Based on this, in the first aspect, the cost evaluation model can be used to determine the target task combination with the lowest query cost from multiple task combinations, thereby adaptively realizing the allocation of stream and batch tasks, and making full use of the resources of the batch and stream integrated data processing framework to perform data queries in a comprehensive and optimal manner in terms of query efficiency and cost, and obtain query results.

[0010] In a possible implementation manner of the first aspect, the multiple subtasks are obtained by splitting the data query task according to at least one of a query statement structure, a query time range of the data query task, and aggregation time information of the data query task.

[0011] In this possible implementation, the data query task can be disassembled based on one or more splitting dimensions including the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task, so that the query load is broken up along one or more dimensions to reduce the granularity of the query load, improve data query efficiency, and facilitate the subsequent adaptive allocation of stream processing or batch processing to each subtask, thereby improving the overall query performance.

[0012] In a possible implementation of the first aspect, a processing method for each subtask is determined based on one or more of the following information: an operator type corresponding to the subtask, a structure of a query statement, and a write status of a data source corresponding to the subtask.

[0013] In this possible implementation, based on the characteristics and constraints of batch processing methods such as continuous queries and stream processing methods such as stream computing, it can be determined that one or more of the operator type corresponding to the subtask, the query statement structure, and the write status of the data source corresponding to the subtask will affect the possible processing methods of the subtask. Therefore, the possible processing methods of the subtask can be determined based on one or more of the above information.

[0014] In a possible implementation of the first aspect, the batch processing mode indicates that the subtasks are converted into continuous query tasks, and the stream processing mode indicates that the subtasks are converted into stream computing tasks; based on the cost evaluation model, a target task combination is determined from multiple task combinations, including: after each subtask in each task combination is converted into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination; according to the query cost corresponding to each task combination, the target task combination is determined from multiple task combinations.

[0015] In this possible implementation, continuous queries are automatically executed queries that periodically calculate and aggregate data and store the results in new tables. During the data query process, continuous queries can reduce query load and improve query efficiency by pre-calculating data. Stream computing is a real-time data processing technology that processes and analyzes data streams in real time to promptly identify and respond to events and trends in the data stream. Stream computing is data-driven, meaning it computes data as it is written, reducing storage costs and improving query efficiency.

[0016] During the cost evaluation process, task conversion can be performed based on the processing method of the subtasks in the task combination, and cost evaluation can be performed after the task conversion to determine the target task combination with the lowest query cost, thereby realizing the adaptive allocation of stream batch tasks such as continuous query tasks and stream computing tasks, and performing data queries in a comprehensive and optimal manner in terms of query efficiency and cost to obtain query results.

[0017] In a possible implementation of the first aspect, performing a data query according to a target task combination to obtain a query result corresponding to the data query task includes: executing a continuous query task or a stream computing task converted from each subtask in the target task combination to obtain a query result corresponding to the data query task.

[0018] In a possible implementation of the first aspect, based on a cost evaluation model, a target task combination is determined from multiple task combinations, including: based on the cost evaluation model, determining the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask processed in batch mode in each task combination includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost, and the query cost of the subtask processed in stream processing mode includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost; according to the query cost of each task combination in the multiple task combinations, a target task combination is determined from the multiple task combinations.

[0019] In this possible implementation, the cost evaluation model can realize a comprehensive cost evaluation of the execution efficiency and execution cost (such as time cost and cost of consumed computing resources, etc.) of each task combination, thereby calculating the comprehensive execution cost of each task combination, so as to screen out the task combination with the best comprehensive execution efficiency and execution cost from multiple task combinations as the target task combination.

[0020] Among them, the cost evaluation method of the subtask whose processing method is batch processing can be different from the cost evaluation method of the subtask whose processing method is stream processing. Specifically, since in the stream processing method, it is usually not necessary to write the data to a storage device such as a disk and then read it, but calculation is performed when writing. Therefore, data scanning can be omitted, that is to say, the query cost of the subtask whose processing method is stream processing does not include the data scanning cost. Since there is no need to scan the data, there is no need to build a primary key index, etc. Therefore, the query cost of the subtask whose processing method is stream processing can include the data grouping cost, but this data grouping cost is different from the above-mentioned index scanning cost.

[0021] A second aspect of the present application provides a data query device that has the function of implementing the method of the first aspect or any possible implementation of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, such as an acquisition module and a processing module.

[0022] A third aspect of the present application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, and the memory of at least one computing device stores computer execution instructions that can be run on the processor. When the computer execution instructions are executed by the processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a computing device cluster, the computing device cluster executes the method as described in the first aspect or any possible implementation of the first aspect.

[0024] In a fifth aspect, the present application provides a computer program product storing one or more computer-executable instructions. The computer program product includes computer-executable instructions. When the computer-executable instructions are executed by a computing device cluster, the computing device cluster executes the method as described in the first aspect or any possible implementation of the first aspect.

[0025] A sixth aspect of the present application provides a chip system, which includes a processor for supporting a computing device cluster to implement the functions involved in the first aspect or any possible implementation of the first aspect. In one possible design, the chip system may also include a memory for storing necessary program instructions and data. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0026] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] FIG1 is a schematic diagram of an embodiment of a data query method provided in an embodiment of the present application;

[0028] FIG2 is an exemplary schematic diagram of splitting a data query task into multiple subtasks according to an embodiment of the present application;

[0029] FIG3a is an exemplary schematic diagram of a continuous query provided by an embodiment of the present application;

[0030] FIG3 b is an exemplary schematic diagram of stream computing provided by an embodiment of the present application;

[0031] FIG4 is an exemplary schematic diagram of a processing method for determining subtasks provided in an embodiment of the present application;

[0032] FIG5 is an exemplary schematic diagram of obtaining a target task combination provided in an embodiment of the present application;

[0033] FIG6 is a schematic diagram of an exemplary structure of a data query system provided in an embodiment of the present application;

[0034] FIG7 is a schematic diagram of an embodiment of a data query device provided in an embodiment of the present application;

[0035] FIG8 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0036] FIG9 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0037] FIG10 is another schematic diagram of the structure of the computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0039] Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0040] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate. This is merely a way of distinguishing objects with the same properties when describing them in the embodiments of this application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0041] Currently, data processing frameworks such as Spark and Flink support batch-stream integrated data query technology to meet various business needs such as stream data processing and batch data processing.

[0042] Batch-stream integrated data query technology integrates the processing of both stream and batch data. This technology can handle both real-time stream data and batch data, improving data processing efficiency and accuracy. This technology is widely used in data warehousing, data analysis, real-time monitoring, and other fields.

[0043] In current practice, when performing data queries, users must manually allocate the stream processing and batch processing tasks required to complete the query within data processing frameworks such as Spark and Flink. This means that current data processing frameworks typically rely on manual experience to allocate task processing methods, obtaining stream and batch tasks and executing them through integrated batch and stream data query technology. Limited by manual experience, this task allocation method struggles to fully utilize the resources within the data processing framework and determine the optimal execution method (e.g., the most cost-effective execution method, considering factors such as efficiency and cost).

[0044] Based on this, an embodiment of the present application provides a data query method that can solve the problem of relying on manual experience to implement the allocation of stream processing tasks and batch processing tasks, which may result in the inability to make good use of the resources of the data processing framework and determine the task execution method with better execution effect.

[0045] The method of the embodiment of the present application can be applied to a computing device cluster, which may include one or more computing devices.

[0046] The type of any computing device is not limited herein. For example, any computing device may be a terminal device, or a server, a container, or a virtual machine.

[0047] In embodiments of the present application, the computing device cluster can implement integrated batch and stream data queries. For example, a data processing framework such as Spark or Flink can be deployed in the computing device cluster. Alternatively, the computing device cluster can be connected to other devices that are deployed with data processing frameworks such as Spark or Flink to utilize the data processing frameworks in the other devices to implement integrated batch and stream data queries.

[0048] In one example, the computing device cluster can be used to implement a cloud platform. In this case, the embodiments of the present application can be applied to the cloud platform.

[0049] The cloud platform can provide computing, network and storage capabilities based on services of hardware resources and software resources. Exemplarily, the cloud platform can include one or more data centers to provide cloud resources through one or more data centers.

[0050] In an exemplary application scenario of a cloud platform, a cloud platform implemented by a cluster of computing devices can provide users with cloud services that can execute the methods of the embodiments of the present application, so that users can purchase or obtain the cloud services to implement data queries.

[0051] Based on the computing device cluster, as shown in FIG1 , a data query method in an embodiment of the present application includes steps 101 - 104 .

[0052] Step 101: Obtain a query statement.

[0053] Query statements are used to issue data query tasks.

[0054] The query statement may adopt Structured Query Language (SQL), or may adopt other existing or subsequently developed query languages.

[0055] The specific source of the query statement is not restricted here.

[0056] In one example, the query statement is received by the computing device cluster through interaction with a user or a device deployed on the user's side. For example, a user-side device may send a query instruction containing the query statement to the computing device cluster, causing the computing device cluster to receive the query statement and trigger the execution of a corresponding data query task. Alternatively, a user may input the query statement into the computing device cluster, causing the computing device cluster to receive the query statement and trigger the execution of a corresponding data query task.

[0057] In another example, in a big data scenario, the computing device cluster may be a computing device cluster including multiple computing nodes. During the data analysis and processing, the computing device cluster generates the query statement and triggers the execution of the corresponding data query task.

[0058] A data query task refers to obtaining query results from a specified data source based on a query statement. The data source can be located in a computing device cluster executing the embodiment of the present application, or stored in other devices such as a database.

[0059] The specific type of the data query task is not limited herein. Exemplarily, the data query task may be a data query on stream data and / or batch data.

[0060] Step 102: Determine multiple task combinations.

[0061] Among them, each task combination includes multiple subtasks, and the multiple subtasks are obtained by splitting the data query task. The processing method of each subtask is batch processing or stream processing. Among them, the multiple task combinations include a first task combination and a second task combination, and the first task combination and the second task combination include at least one different subtask, or, the multiple task combinations include a third task combination and a fourth task combination, and the multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing methods of the multiple subtasks in the third task combination and the combination of the processing methods of the multiple subtasks in the fourth task combination are different.

[0062] In the embodiment of the present application, the process of determining multiple task combinations may include the following processing stages:

[0063] Split the data query task, determine the feasible processing method for each subtask, and combine the subtasks to obtain multiple task combinations.

[0064] The following is an illustrative introduction to each processing stage in the process.

[0065] A. Split data query tasks.

[0066] There are many specific ways to split the data query task, which are not limited here.

[0067] In some examples, the multiple subtasks are obtained by splitting the data query task according to at least one of the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task.

[0068] In this way, the data query task can be disassembled based on one or more splitting dimensions including the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task, so that the query load is broken up along one or more dimensions to reduce the granularity of the query load, improve data query efficiency, and facilitate the subsequent adaptive allocation of stream processing or batch processing to each subtask, thereby improving the overall query performance.

[0069] Below, several possible splitting dimensions and corresponding splitting methods are introduced by way of example.

[0070] 1. The structure of the query statement.

[0071] In some examples, the data query task can be split based on the splitting dimension of the query statement structure.

[0072] For example, by parsing the structure of the query statement, it can be determined that the query statement includes three levels of nested subqueries. Then, based on the query level and the splitting dimension of the query statement structure, the data query task can be split into three subtasks.

[0073] 2. The time range of the data query task.

[0074] The time range of a data query task can include the time period to be queried by the data query task. For example, if the query statement indicates to query the specified data within 3 days from January 1, 2023 to January 3, 2023, then the time range of the data query task is 3 days. In this example, based on the splitting dimension of the time range of the data query task, the data query task can be split according to smaller time granularity or time interval, and the data query task can be split within the time range. For example, splitting the day into 3 subtasks, each of which corresponds to a day within the 3-day time period.

[0075] 3. Aggregation time information of data query tasks.

[0076] The aggregation time information of the data query task may include the duration of the aggregation time window of the data query task. Generally speaking, the aggregation time window of the data query task may be a time window for performing aggregation operations within the time range of the data query task. For example, the query instruction indicates to query the specified data within 3 days from January 1, 2023 to January 3, 2023, and the aggregation time information of the data query task indicates to perform aggregation operations within the 3-day period with an aggregation time window of 1 hour. Then, in this example, the aggregation time information of the data query task may be 1 hour. In this example, based on the splitting dimension of the aggregation time information of the data query task, the data query task may be split into multiple subtasks within the preset 1-hour aggregation time window according to a smaller time granularity, such as 1 minute, 10 minutes, 30 minutes, etc. It can be understood that the aggregation time of the data query task is a finer-grained time window within the time range of the data query task.

[0077] It can be seen that by splitting the data query task based on time information such as the query time range of the data query task and the aggregation time information of the data query task, the query load of the current data query task can be broken up along the time dimension. In the time dimension, the granularity of the query load is reduced, thereby reducing the query load within the unit time range. In this way, under the conditions of limited computing resources, query operations such as high-load and big data queries can be efficiently executed, thereby reducing the resource cost and time cost of data queries and improving data query efficiency.

[0078] For example, as shown in FIG2 , it is an exemplary schematic diagram of splitting a data query task along multiple splitting dimensions.

[0079] After obtaining the query instruction carrying the query statement, the data query task is split according to three splitting dimensions: the structure of the query statement, the time range of the data query task, and the aggregation time of the data query task.

[0080] Among them, in the structure of the query statement, it is determined that the query statement SQL includes three layers of nested subqueries, namely level1, level2 and level2, and the data query task is divided into three subtasks: SQL (level1), SQL (level2) and SQL (level3) in the structure of the query statement.

[0081] In the time range corresponding to the data query task, since the query instruction indicates to query the specified data within the three days of day1, day2 and day3, the data query task can be split into three subtasks: SQL (day1), SQL (day2), and SQL (day3) within this time range.

[0082] In the aggregation time corresponding to the data query task, the aggregation time can be 1 hour (1 hour), 2 hours (2 hours) and 3 hours (3 hours). Then, the data query task can be split into three subtasks of SQL (1 hour), SQL (2 hours) and SQL (3 hours) within this time range.

[0083] B. Determine feasible handling methods for each subtask.

[0084] After splitting the data query task into multiple subtasks, you can determine the possible processing methods for each subtask.

[0085] Each subtask may be processed in batch mode and / or stream mode.

[0086] That is, for a subtask, it can be determined, for example, through conversion rules, whether the subtask can be executed in batch processing, or in stream processing, or both.

[0087] When a subtask is processed in a batch processing mode, the subtask can be processed using batch processing resources and technologies (such as batch processing resources and technologies in a data processing engine).

[0088] The specific content of the batch processing method is not limited here. The batch processing method can be an existing batch processing method or a batch processing method developed subsequently.

[0089] For example, a batch processing method may be continuous query (CQ).

[0090] A continuous query is an automatically executed query that periodically calculates and aggregates data and stores the results in a new table. During the data query process, continuous queries can reduce query load and improve query efficiency by precalculating data.

[0091] In the example shown in Figure 3a, before the continuous query, the data source is usually pre-stored in the underlying storage device. When defining a continuous query, it is necessary to specify the time window size and the forward sliding times for each forward increment. During the continuous query process, after each time window is closed and the next time window begins, the data in the just-closed time window is calculated (for example, performing scheduled calculations, aggregating data, etc.), and the calculation results are saved, for example, and then stored in the storage device. Therefore, the overall process of continuous query is to query data from the bottom up and then write the corresponding calculation results from the top down. The query load can be reduced and the query efficiency can be improved by calculating the data of each time window in advance.

[0092] In some other examples, the batch processing method may also include other processing methods besides continuous query, for example, it may include one or more processing methods such as pre-aggregation and downsampling.

[0093] When the processing mode of the subtask is stream processing, the subtask can be processed by using stream processing resources and technologies (such as stream processing resources and technologies in a data processing engine).

[0094] The specific content of the stream processing method is not limited here. The stream processing method may be an existing stream processing method or a subsequently developed stream processing method.

[0095] For example, one stream processing method may be stream computing.

[0096] Stream computing is a real-time data processing technology that processes and analyzes data streams in real time, enabling timely discovery and response to events and trends within the data stream. Stream computing is data-driven, meaning it computes as it's written, reducing storage costs and improving query efficiency.

[0097] In the example shown in Figure 3b, an exemplary data transmission process for stream computing can be: after receiving a real-time data stream, through information processing and interaction such as filtering, local aggregation (LocalAgg), global aggregation (GlobaAgg), and stream management modules, the stream computing results are obtained and stored through output operations such as flush, for example, storing the stream computing results to an underlying storage device such as a disk. As can be seen, the overall process of stream computing is top-down, with computing performed while writing data.

[0098] In the embodiment of the present application, optional batch processing methods (such as continuous query, pre-aggregation, downsampling, etc.) and optional stream processing methods (such as stream computing, etc.) can be predetermined.

[0099] After obtaining multiple subtasks, it can be determined based on the conversion rules whether each subtask is suitable for batch processing or stream processing, thereby determining a feasible processing method for each subtask.

[0100] In addition, in some examples, there can be more than one batch processing mode, and / or there can be more than one stream processing mode. For example, in the conversion rules, the optional batch processing modes include batch processing mode 1 and batch processing mode 2, and the optional stream processing mode includes stream processing mode 1. For subtask B, if it is determined according to the conversion rules that subtask B is suitable for both batch processing mode 1 and batch processing mode 2 as well as stream processing mode 1, then subtasks B1, B2, and B3 of subtask B can be obtained. Among them, the processing mode corresponding to subtask B1 is batch processing mode 1, the processing mode corresponding to subtask B2 is batch processing mode 2, and the processing mode of subtask B2 is stream processing mode 1.

[0101] In the embodiment of the present application, the specific content of the conversion rule for determining the feasible processing mode of the subtask is not limited here. For example, the conversion rule can be pre-configured by the developer based on the characteristics and constraints of the batch processing mode and the stream processing mode.

[0102] The following uses continuous query as a batch processing method and stream computing as a stream processing method as an example to illustrate the characteristics and constraints of continuous query and stream computing.

[0103] Continuous queries can have the following features: 1. Support high-order operators and complex queries; 2. Support distributed scheduling and load balancing.

[0104] Continuous queries may have the following constraints: 1. During a continuous query, large amounts of data may result in high disk input / output (I / O) and network overhead. 2. If multiple continuous query tasks execute queries on the same table, each continuous query task must read the table separately, resulting in repeated reading of data from the table, which may lead to timing conflicts, delays, and wasted resource overhead.

[0105] Stream computing has the following characteristics: 1. It performs calculations through memory, which is fast and has little impact on storage; 2. It has high computing efficiency and low disk I / O and network overhead.

[0106] Stream computing can have the following constraints: 1. The cache will reside in memory, which may increase memory overhead; 2. It only supports incremental computing operators, such as sum, count, min, and max.

[0107] In this way, based on the characteristics and constraints of batch processing methods such as continuous query and stream processing methods such as stream computing, transformation rules can be configured to determine feasible processing methods for subtasks through the transformation rules.

[0108] In some embodiments, the processing manner of each subtask is determined based on one or more of the following information:

[0109] The operator type corresponding to the subtask, the structure of the query statement, and the write status of the data source corresponding to the subtask.

[0110] The following is an exemplary introduction of each.

[0111] 1. Operator type corresponding to the subtask.

[0112] In some examples, the feasible processing mode for a subtask can be determined based on the operator type corresponding to the subtask. For example, if the operator type corresponding to the subtask can be executed via batch processing, then one feasible processing mode for the subtask is batch processing. If the operator type corresponding to the subtask can also be executed via stream processing, then another feasible processing mode for the subtask is stream processing.

[0113] For example, the operator corresponding to a subtask is an aggregation operator. Since the aggregation operator does not support incremental computing, it cannot be implemented through stream processing, but can only be implemented through batch processing methods such as continuous query. In this case, the subtask only corresponds to batch processing, which is a feasible processing method.

[0114] 2. The structure of the query statement.

[0115] The structure of query statements may vary across different query scenarios. Therefore, the relationships between subtasks resulting from splitting a data query task based on the query statement's structure may differ. For example, in some scenarios, the subtasks resulting from splitting a data query task based on the query statement's structure may be nested or parallel and independent. Because the relationships between subtasks influence the possible processing methods for these subtasks, the query statement's structure can be used to determine the possible processing methods for these subtasks.

[0116] For example, if the data query task is split according to the structure of the query statement, three subtasks are obtained, and according to the structure of the query statement, it can be determined that among the three subtasks, the input data of subtask B is the output data of subtask A, and subtask B can be driven by the output data of subtask A, then this subtask B can adopt stream processing methods such as stream computing.

[0117] 3. The write status of the data source corresponding to the subtask.

[0118] The data source corresponding to a subtask refers to the data object that the subtask processes or queries. The storage location of this data source can vary in different processing scenarios.

[0119] For example, in some scenarios, the data source corresponding to a subtask may have been pre-stored on a storage device such as a hard drive within the computing cluster, or on a storage device external to the cluster. Based on this write status, the subtask can be processed using batch processing methods such as continuous queries.

[0120] In other scenarios, the data source corresponding to a subtask may not be written to a storage device such as a hard drive. In this case, the data source may be a real-time input data stream in a real-time scenario, that is, streaming data, which can be stored in memory. Since the data source is not written to a storage device such as a hard drive, the corresponding subtask can be processed using stream processing methods such as Stream Compute.

[0121] After splitting the data query task into multiple subtasks and determining the possible processing methods for each subtask, multiple task combinations can be obtained based on the multiple subtasks and the possible processing methods for each subtask. Each task combination can include at least two subtasks, and each subtask can be processed in batch or stream mode.

[0122] For example, FIG4 is an exemplary schematic diagram for determining a feasible processing method for each subtask.

[0123] In the example shown in Figure 4, based on the structure of the query statement, it is determined that the query statement SQL includes three layers of nested subqueries, namely level1, level2 and level2. Then, based on the structure of the query statement, the data query task is split into three subtasks: SQL1 (level1), SQL1 (level2), and SQL1 (level3).

[0124] According to the conversion rules, the operator corresponding to subtask SQL1 (level 1) cannot use stream processing methods such as batch computing, but can only use stream processing. Therefore, the feasible processing method for subtask SQL1 (level 1) is stream processing. Furthermore, according to the conversion rules, the operator corresponding to subtask SQL (level 2) can only use batch processing and cannot use stream processing. Therefore, the feasible processing method for subtask SQL (level 2) is batch processing. Furthermore, according to the conversion rules, the operator corresponding to subtask SQL (level 3) can use both stream processing and batch processing. Therefore, the feasible processing methods for subtask SQL (level 3) are batch processing and stream processing.

[0125] C. Combine subtasks to obtain multiple task combinations.

[0126] After determining feasible processing methods for each subtask, multiple task combinations can be obtained.

[0127] Among them, each task combination includes multiple subtasks, and the multiple subtasks are obtained by splitting the data query task. The processing method of each subtask is batch processing or stream processing. Among them, the multiple task combinations include a first task combination and a second task combination, and the first task combination and the second task combination include at least one different subtask, or, the multiple task combinations include a third task combination and a fourth task combination, and the multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing methods of the multiple subtasks in the third task combination and the combination of the processing methods of the multiple subtasks in the fourth task combination are different.

[0128] In the embodiment of the present application, each task combination can complete the data query task.

[0129] There are differences between different task combinations. Specifically, there may be differences in subtasks and / or there may be the same subtasks but the same subtasks may be processed in different ways.

[0130] The following example illustrates this.

[0131] In one example, the multiple task combinations include a first task combination and a second task combination, and subtasks of the first task combination and subtasks of the second task combination may differ. In addition, in some examples, when the first task combination and the second task combination include the same subtasks, the same subtasks may be processed in the same or different manners.

[0132] For example, in one example, the first task combination and the second task combination are split in different ways.

[0133] For example, when the structure of the query statement is nested, the data query task is split only according to the structure of the query statement to obtain three subtasks, namely subtask A, subtask B and subtask C; based on this splitting method, the first task combination is obtained, and the first task combination includes subtask A, subtask B and subtask C.

[0134] In addition, the data query task can also be split according to the structure of the query statement. After obtaining subtasks A, subtask B and subtask C, subtask B can be split into subtask B1 and subtask B2 according to the time range of the data query task, and subtask C can be split into subtask C1 and subtask C2. Then, a second task combination can be obtained according to this splitting method. Specifically, the second task combination includes 5 subtasks, namely subtask A, subtask B1, subtask B2, subtask C1 and subtask C2.

[0135] It can be seen that, in this example, the first task combination and the second task combination include at least one different subtask.

[0136] In addition, among the multiple task combinations, there may be any two task combinations that have the same subtasks, but the same subtasks are processed in different ways.

[0137] Specifically, the multiple task combinations include a third task combination and a fourth task combination. The multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of processing methods of the multiple subtasks in the third task combination and the combination of processing methods of the multiple subtasks in the fourth task combination are different.

[0138] For example, when the query statement structure is nested, the data query task is split only according to the query statement structure to obtain three subtasks, namely subtask A, subtask B, and subtask C.

[0139] Then, it can be determined that the feasible processing modes for subtask A are batch processing and stream processing, the feasible processing modes for subtask B are batch processing and stream processing, and the feasible processing mode for subtask C is batch processing.

[0140] In this way, four task combinations can be obtained in this example. The multiple subtasks in the four task combinations are the same, but the combinations of the corresponding processing methods of the multiple subtasks are different.

[0141] For example, the four task combinations may include a third task combination, which includes subtask A, subtask B and subtask C, and the processing method of subtask A in the third task combination is batch processing, the processing method of subtask B is batch processing, and the processing method of subtask C is batch processing.

[0142] Furthermore, the four task combinations may include a fourth task combination, which includes subtask A, subtask B and subtask C. Moreover, the processing method of subtask A in the fourth task combination is stream processing, the processing method of subtask B is batch processing, and the processing method of subtask C is batch processing.

[0143] It can be seen that in this example, the processing method of subtask A in the third task combination is different from the processing method of subtask A in the fourth task combination.

[0144] Step 103 : Based on the cost evaluation model, a target task combination is determined from the multiple task combinations, where the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations.

[0145] In some examples, a cost evaluation model can be used to perform cost evaluation on each of multiple task combinations to obtain a task combination with the lowest query cost as a target task combination.

[0146] In other examples, when there are a large number of task combinations, these large number of task combinations can be preliminarily screened through evolutionary algorithms (such as genetic algorithms or particle swarm algorithms) to delete task combinations with significantly higher query costs. Then, a cost evaluation model can be used to determine the target task combination from the preliminarily screened task combinations.

[0147] This cost evaluation model can realize a comprehensive cost evaluation of the execution efficiency and execution cost (such as time cost and the cost of consumed computing resources, etc.) of each task combination, thereby calculating the comprehensive execution cost of each task combination, so as to screen out the task combination with the best comprehensive execution efficiency and execution cost from multiple task combinations as the target task combination.

[0148] The specific content of the cost evaluation model can be various. For example, the cost evaluation method for a batch processing subtask can be different from the cost evaluation method for a stream processing subtask. Specifically, the cost evaluation can be performed based on the subtask processing method.

[0149] In some embodiments, during the cost evaluation process, task conversion may be performed according to the processing method of subtasks in the task combination, and cost evaluation may be performed after the task conversion.

[0150] Specifically, in some embodiments, the batch processing mode indicates converting the subtask into a continuous query task, and the stream processing mode indicates converting the subtask into a stream computing task;

[0151] Based on the cost evaluation model, the target task combination is determined from multiple task combinations, including:

[0152] After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination.

[0153] According to the query cost corresponding to each task combination, the target task combination is determined from multiple task combinations.

[0154] In an embodiment of the present application, after each subtask in each task combination is converted into a continuous query task or a stream computing task, the query cost corresponding to each task combination can be determined based on the cost evaluation method of the continuous query task and the cost evaluation method of the stream processing task.

[0155] Below, in the cost evaluation model, the cost evaluation method for subtasks whose processing method is a batch processing method such as continuous query and the cost evaluation method for subtasks whose processing method is a stream processing method such as stream computing are respectively introduced by way of example.

[0156] 1. Cost evaluation method for subtasks in batch processing mode.

[0157] In one embodiment, the query cost of a subtask processed in batch mode includes at least one of the following cost evaluation indicators:

[0158] Index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost.

[0159] The following describes each cost evaluation indicator with specific examples.

[0160] In an exemplary scenario A, a subtask that adopts a batch processing method adopts a primary key index method.

[0161] The timeline of this subtask is M, involving n indexes (indexs). The cardinality (number of distinct elements) of the i-th index is card(indexs[i]), the aggregate field cardinality is I, and the primary key cardinality is P.

[0162] The number of rows of data to be queried is N, and the average number of rows for each primary key is N / P.

[0163] The number of data files is F, and the number of files to be queried is Fq, among which the number of ordered files is Fq1, the number of unordered files is Fq2, and the file selection rate is Fq / F.

[0164] The number of ordered rows for each primary key is (N / P) / Fq*Fq1, and the number of out-of-order rows for each primary key is (N / P) / Fq*Fq2.

[0165] In this scenario, each file uses k hash functions to generate a bloom filter. The data layout in each file is divided into different chunks according to the timeline. Each chunk has corresponding metadata (called chunkMeta), and every 1024 chunkMeta corresponds to a metadata index (metaIndex).

[0166] The ratio of the amount of data filtered by the query to the total amount of data is called the selectivity. In this scenario, if a multi-timeline spatiotemporal aggregation query is implemented using technologies such as Open Gemini, without filtering on the timeline dimension, the selectivity is M / M, almost 100%. If the number of background timelines is M and the number of timelines selected by the query is S, the corresponding selectivity is S / M. The amount of data to be queried is N. The number of spatial and temporal groups is Gs and Gt, respectively, and the total number of groups G = Gs * Gt.

[0167] In the cost assessment model, the algorithmic complexity corresponding to one or more cost assessment metrics can be considered. For example, in the scenario of multi-timeline spatiotemporal aggregation queries, multiple aspects such as indexing, data scanning, data calculation, data conversion, and data transmission all involve algorithmic complexity overhead. The complexity coefficients corresponding to these various aspects can vary, but are uniformly represented by the C (correlation coefficient).

[0168] In this scenario, the specific calculation method for the cost evaluation index corresponding to the subtasks processed in batch mode is as follows:

[0169] 1) Index scan cost.

[0170] The index scan cost can reflect the resource cost of the index scan operation corresponding to the corresponding subtask, such as time cost and computing resource overhead, etc. The index scan cost may be related to the indexing method, related indexing algorithm, etc.

[0171] For example, in scenario A above, the index scan cost is calculated using the following terms:

[0172] Among them, considering that the primary key groups will be traversed during index scanning and the order between groups and within groups must be maintained, the cost of the index scan is related to the primary key cardinality P and the complexity of the group calculation.

[0173] Specifically, in this scenario, the grouping computation complexity is calculated by To describe.

[0174] 2) Data scanning cost.

[0175] The overhead of scanning data is reflected in filtering files through Bloom filters, finding the corresponding chunks in each file through binary search, and scanning data in each chunk.

[0176] In the above scenario A, the data scanning cost is calculated by the following terms:

[0177] The following explains the meaning of each of the above items:

[0178] P×F q ×k: Determine whether each primary key is in the file;

[0179] Determine whether the data block of each primary key is in the file;

[0180] Scan data from sorted and unsorted files.

[0181] 3) Data computation cost.

[0182] Data computation costs may include, for example, aggregation computation costs.

[0183] Regarding the aggregation computation cost, the main overhead of spatial aggregation is the number of rows N of data to be queried. For example, in a system acceleration query computation technology, considering the vectorization computation coefficient v of vectorization, the data computation cost can be calculated by the following terms:

[0184] Among them, C is the coefficient corresponding to the cost of calculating the data.

[0185] 4) Data conversion cost.

[0186] In some examples, devices such as upper-layer clients may have requirements for the data format, such as order, which may require data format conversion, such as sorting, which incurs corresponding overhead. Based on this, the cost evaluation metric may include data conversion cost, specifically, sorting cost, etc.

[0187] Among them, for the sorting cost, the main sorting overhead may be the sorting of calculation results among multiple groups.

[0188] For example, in the above scenario A, the calculation result of each group is calculated as 1 value. The data conversion cost can be calculated by the following items:

[0189] Among them, v is the vectorized calculation coefficient of vectorization, and C is the coefficient corresponding to the data conversion cost.

[0190] 5) Data transmission cost.

[0191] Regarding data transmission costs, data transmission overhead can come from the data after aggregation, for example, the overhead involved in reading data from the bottom up.

[0192] In the above scenario A, suppose that spatiotemporal aggregation is performed on n fields (for example, based on the region or other attributes of the hosts in the server cluster) and grouped by a1, a2, ..., an, with a grouping cardinality of card (a i ), the data transmission cost can be calculated as follows:

[0193] Wherein, C is the coefficient corresponding to the data transmission cost.

[0194] 6) Data writing cost.

[0195] Cost of data writing write The main data writing overhead may include the overhead of writing data into storage space such as a hard disk, memory, etc. and / or the overhead of the CPU storing data from the storage space.

[0196] 7) Interface interaction cost.

[0197] Cost of interface interaction IO ,Interface interaction overhead can come from interface interaction operations such as memory interface, disk interface, etc.

[0198] Thus, in the above scenario A, based on the above costs, the cost of the subtasks in batch processing is Cost CQ It can be calculated based on the following formula:

[0199] Of course, in other examples, the cost evaluation metrics for subtasks using batch processing may include one or more of the above examples, and the specific calculation methods of the above cost evaluation metrics may vary based on different scenario requirements. For example, when the indexing method uses a normal index, the specific calculation method of the index scan cost may be different from the above example.

[0200] 2. The query cost of a subtask processed in stream processing mode includes at least one of the following cost evaluation indicators:

[0201] Data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost.

[0202] In particular, since stream processing typically does not require writing data to a storage device such as a disk and then reading it, but rather performs calculations as it is written, data scanning can be omitted. In other words, the query cost for subtasks using stream processing does not include data scanning costs. Since data scanning is not required, there is no need to build primary key indexes, etc. Therefore, the query cost for subtasks using stream processing can include data grouping costs, but this data grouping cost is different from the index scanning cost described above.

[0203] Therefore, for example, in the above scenario A, referring to the example of the subtask in batch processing, the cost value of the subtask in stream processing is SC It can be calculated based on the following formula:

[0204] The data packet cost is

[0205] In this way, under the condition of fixed computing resources, we can determine the optimal combination of batch processing subtasks and stream processing subtasks through cost evaluation, obtain the target task combination, and rationally utilize resources to achieve efficient data query and improve query performance.

[0206] For example, in the example shown in Figure 5, the data query task is divided into three subtasks based on the structure of the query statement, namely subtask SQL (level 1), subtask SQL (level 2), and subtask SQL (level 3). The three subtasks are subqueries nested in three levels.

[0207] Taking the three subtasks in the query statement structure as an example, the possible forms of target task combinations are introduced.

[0208] In the example shown in FIG5 , based on the cost evaluation model, it can be seen that the subtask SQL (level 1) has better processing performance when the stream processing method is used. Therefore, in the target task combination, the subtask SQL (level 1) is processed by the stream processing method.

[0209] In addition, the processing performance of subtask SQL (level 2) is better when it is processed in batch mode. Therefore, in the target task combination, subtask SQL (level 2) is processed in batch mode. In this way, batch processing is adaptively implemented from subtask SQL (level 1) to subtask SQL (level 2).

[0210] Due to the nested relationship, the processing performance of the process of implementing the calculation process of subtask SQL (level 3) by using the calculation data writing of subtask SQL (level 2) as the driver is better. Therefore, in the target task combination, subtask SQL (level 3) is processed using stream processing, and the calculation data written by subtask SQL (level 2) using batch processing can be used as the input stream data of subtask SQL (level 3).

[0211] It can be seen that in this example, the appropriate processing method for each subtask in the target task combination can be adaptively determined based on the cost evaluation model, and in this example, the stream-batch conversion can be realized in the target task combination determined by the cost evaluation model, thereby making full use of the resources of the batch-stream integrated data query system to execute data query tasks, so as to improve data query efficiency.

[0212] Step 104: perform data query according to the target task combination to obtain query results corresponding to the data query task.

[0213] In some examples, the batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task. Then, the continuous query task or stream computing task after each subtask in the target task combination is executed to obtain the query result corresponding to the data query task.

[0214] In an embodiment of the present application, after determining the target task combination, an execution plan can be constructed based on the continuous query task and / or stream computing task in the target task combination using a directed acyclic graph (DAG) to perform task scheduling and execution, thereby obtaining the query result corresponding to the data query task.

[0215] In an embodiment of the present application, after splitting the data query task, the feasible processing method for each subtask can be adaptively determined, thereby obtaining multiple task combinations based on the feasible processing method of each subtask. Among them, the processing method of the subtasks in the task combination will affect the query cost when executing the data query task based on the task combination. In other words, under different combinations of multiple subtasks and corresponding processing methods, the comprehensive cost of the execution efficiency and execution cost of the data query task is often different.

[0216] Based on this, in the embodiment of the present application, the cost evaluation model can be used to determine the target task combination with the lowest query cost from multiple task combinations, thereby adaptively realizing the allocation of stream and batch tasks, and making full use of the resources of the batch and stream integrated data processing framework to perform data queries in a comprehensive and optimal manner in terms of query efficiency and cost, and obtain query results.

[0217] Any of the above data query method embodiments may be implemented by one or more functional modules in a data query system deployed in a computing device cluster.

[0218] In conjunction with any of the above embodiments, the data query system that can be deployed in the computing device cluster is exemplarily introduced below.

[0219] For example, as shown in FIG6 , it is a schematic diagram of an exemplary framework of a data query system implemented based on the computing device cluster.

[0220] The data query system may include one or more of the multiple functional modules shown in FIG6 to implement any of the above-mentioned data query method embodiments.

[0221] The following is an exemplary introduction to the functional modules that may be included in the data query system.

[0222] In the data query system, interfaces such as query statements such as structured query language (SQL), data streams and / or machine learning (ML) application program interfaces (APIs) can be provided as entry points for query instructions carrying query statements, data streams, and data analysis information based on machine learning.

[0223] In addition, the data query system may include one or more of a parser, a task identification module, a task decomposition module, a rule module, a cost module, a DAG construction module, a DAG scheduling module, and a DAG execution module.

[0224] Specifically, the parser can parse query statements, etc., to identify the content of the data query task. For example, the query statement can be parsed to obtain a query syntax tree, and then a query object can be generated through syntactic and semantic analysis to obtain a parsing result. The parsing result can then be transmitted to the task identification module to identify the type of data query task and mark the type of the data query task. The specific type of the data query task is not specifically limited here.

[0225] After the data query task is identified, the data query task can be decomposed by a task decomposition module to obtain corresponding subtasks.

[0226] The rule module then determines feasible subtask processing methods based on the transformation rules and obtains multiple task combinations. The cost module then uses the cost evaluation model to obtain the cost value of each task combination, thereby determining the target task combination with the optimal cost from the multiple task combinations.

[0227] After obtaining the target task combination, an execution plan can be constructed through the DAG construction module. The task is then scheduled through the DAG scheduling module and executed through one or more executors in the DAG execution module to obtain the query results corresponding to the data query task.

[0228] The data query system may be applied in a variety of scenarios, that is, any of the above-mentioned data query method embodiments of the present application may be applied in a variety of scenarios, which are not limited here.

[0229] In some examples, the data query system can be used for database query, and can also be used in scenarios such as big data management. Moreover, the specific functions of the data query system can be determined based on the scenario.

[0230] For example, the method of decomposing a data query task to obtain multiple subtasks in any of the above embodiments can be applied to a rule-based optimizer (RBO) in a data query system to decompose queries with large data volumes along one or more splitting dimensions such as the time dimension, thereby dispersing large query loads along the time dimension and reducing overall computing resource overhead.

[0231] In addition, the method for performing task combination evaluation based on the cost evaluation model in any of the above embodiments can be applied to a cost-based query optimizer (CBO) in a data query system to perceive the query load and adaptively adjust stream and batch computing tasks to improve query efficiency.

[0232] The data query method provided by the embodiment of the present application has been described above from multiple aspects. The data query device provided by the embodiment of the present application will be described below with reference to the accompanying drawings.

[0233] As shown in FIG7 , an embodiment of the present application provides a data query device 70 , which includes:

[0234] The acquisition module 701 is used to acquire a query statement, which is used to issue a data query task;

[0235] The processing module 702 is configured to:

[0236] Determine multiple task combinations, where each task combination includes multiple subtasks, the multiple subtasks are obtained by splitting the data query task, and each subtask is processed in a batch processing manner or a stream processing manner, wherein the multiple task combinations include a first task combination and a second task combination, the first task combination and the second task combination include at least one different subtask, or the multiple task combinations include a third task combination and a fourth task combination, the multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of processing methods for the multiple subtasks in the third task combination and the combination of processing methods for the multiple subtasks in the fourth task combination are different;

[0237] Based on the cost evaluation model, a target task combination is determined from multiple task combinations. The query cost of the target task combination is lower than that of other task combinations in the multiple task combinations.

[0238] Perform data query based on the target task combination to obtain the query results corresponding to the data query task.

[0239] Optionally, the multiple subtasks are obtained by splitting the data query task according to at least one of a query statement structure, a query time range of the data query task, and aggregation time information of the data query task.

[0240] Optionally, the processing mode of each subtask is determined based on one or more of the following information:

[0241] The operator type corresponding to the subtask, the structure of the query statement, and the write status of the data source corresponding to the subtask.

[0242] Optionally, the batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task;

[0243] The processing module 702 is used to:

[0244] After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination.

[0245] According to the query cost corresponding to each task combination, the target task combination is determined from multiple task combinations.

[0246] Optionally, the processing module 702 is configured to execute the continuous query task or stream computing task converted from each subtask in the target task combination to obtain a query result corresponding to the data query task.

[0247] Optionally, the processing module 702 is configured to:

[0248] Based on the cost evaluation model, determine the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask in each task combination processed in batch mode includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost; the query cost of the subtask in each task combination processed in stream mode includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost;

[0249] A target task combination is determined from the multiple task combinations according to the query cost of each task combination in the multiple task combinations.

[0250] In an embodiment of the present application, a module is used as an example of a software functional unit, and the data query device may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the above-mentioned computing device may be one or more. For example, the data query device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same available zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0251] Similarly, the multiple hosts / virtual machines / containers used to run this code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is located within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0252] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, the data query device may be implemented using an application specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex PLD (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0253] The multiple computing devices included in the data query device can be distributed in the same region or in different regions. The multiple computing devices included in the data query device can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data query device can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0254] It should be noted that, in other embodiments, the acquisition module can be used to execute any step in the data query method, and the processing module can be used to execute any step in the data query method. The steps that the acquisition module and the processing module are responsible for implementing can be specified as needed. The full functions of the data query device are realized by respectively implementing different steps in the data query method through the acquisition module and the processing module.

[0255] The present application also provides a computing device 80. As shown in FIG8 , computing device 80 includes a bus 82, a processor 84, a memory 86, and a communication interface 88. Processor 84, memory 86, and communication interface 88 communicate with each other via bus 82. Computing device 80 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 80.

[0256] Bus 82 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 84 may include a pathway for transmitting information between various components of computing device 80 (e.g., memory 86, processor 84, and communication interface 88).

[0257] The processor 84 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0258] The memory 86 may include volatile memory, such as random access memory (RAM). The processor 84 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0259] The memory 86 stores executable program code, and the processor 84 executes the executable program code to respectively implement the functions of the aforementioned acquisition module and training module, thereby implementing the data query method applied to the computing device cluster in the above-mentioned embodiment, that is, the memory 86 stores instructions for executing the data query method applied to the computing device cluster in the above-mentioned embodiment.

[0260] The communication interface 88 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 80 and other devices or a communication network.

[0261] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0262] As shown in Figure 9, the computing device cluster includes at least one computing device 80. The memory 86 in one or more computing devices 80 in the computing device cluster may store the same instructions for executing the data query method.

[0263] In some possible implementations, the memory 86 of one or more computing devices 80 in the computing device cluster may also store some instructions for executing the data query method. In other words, the combination of one or more computing devices 80 can jointly execute the instructions for executing the data query method.

[0264] It should be noted that the memory 86 in different computing devices 80 in the computing device cluster can store different instructions, each for executing a portion of the functions of the data query method. In other words, the instructions stored in the memory 86 in different computing devices 80 can implement the functions of one or more modules in the acquisition module and the training module.

[0265] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), etc. FIG10 illustrates a possible implementation. As shown in FIG10 , two computing devices 80A and 80B are connected via a network. Specifically, the connection to the network is made via a communication interface in each computing device. In this type of possible implementation, the memory 86 in the computing device 80A may store instructions for executing the functions of the acquisition module. Simultaneously, the memory 86 in the computing device 80B may store instructions for executing the functions of the processing module. Alternatively, the memory 86 in the computing device 80A may store instructions for executing part of the functions of the processing module. Simultaneously, the memory 86 in the computing device 80B may store instructions for executing another part of the functions of the processing module.

[0266] It should be understood that the functionality of the computing device 80A shown in FIG10 may also be performed by multiple computing devices 80. Similarly, the functionality of the computing device 80B may also be performed by multiple computing devices 80.

[0267] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection relationship between the computing device clusters in Figures 9 and 10. However, the memory 86 in one or more computing devices 80 in this computing device cluster can store the same instructions for executing the data query method.

[0268] In some possible implementations, the memory 86 of one or more computing devices 80 in the computing device cluster may also store some instructions for executing the data query method. In other words, the combination of one or more computing devices 80 can jointly execute the instructions for executing the data query method.

[0269] It should be noted that the memory 86 in different computing devices 80 in the computing device cluster can store different instructions for executing partial functions of the data query method. In other words, the instructions stored in the memory 86 in different computing devices 80 can implement the functions of one or more modules in the acquisition module and the training module.

[0270] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data query method.

[0271] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data query method.

[0272] The present application also provides a chip system, which includes a processor configured to implement the steps performed by the computing device cluster. In one possible design, the chip system may also include a memory configured to store necessary program instructions and data. The chip system may be composed solely of a chip or may include a chip and other discrete components.

[0273] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0274] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0275] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0276] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0277] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A data query method, characterized in that: The method comprises: Obtaining a query statement, wherein the query statement is used to issue a data query task; Determine a plurality of task combinations, wherein each task combination includes a plurality of subtasks, the plurality of subtasks are obtained by splitting the data query task, and the processing mode of each subtask is a batch processing mode or a stream processing mode, wherein the plurality of task combinations include a first task combination and a second task combination, the first task combination and the second task combination include at least one different subtask, or the plurality of task combinations include a third task combination and a fourth task combination, the plurality of subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing modes of the plurality of subtasks in the third task combination and the combination of the processing modes of the plurality of subtasks in the fourth task combination are different; Based on the cost evaluation model, determining a target task combination from the multiple task combinations, wherein the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations; Perform data query according to the target task combination to obtain query results corresponding to the data query task.

2. The method according to claim 1, characterized in that: The multiple subtasks are obtained by splitting the data query task according to at least one of the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task.

3. The method according to claim 1 or 2, characterized in that: The processing method of each subtask is determined according to one or more of the following information: The operator type corresponding to the subtask, the structure of the query statement, and the writing status of the data source corresponding to the subtask.

4. The method according to any one of claims 1 to 3, characterized in that The batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task; The step of determining a target task combination from the plurality of task combinations based on the cost evaluation model includes: After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination; The target task combination is determined from the multiple task combinations according to the query cost corresponding to each task combination.

5. The method according to claim 4, characterized in that The performing of data query according to the target task combination to obtain a query result corresponding to the data query task includes: Execute the continuous query task or stream computing task converted from each subtask in the target task combination to obtain the query result corresponding to the data query task.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining a target task combination from the plurality of task combinations based on the cost evaluation model includes: Based on the cost evaluation model, determine the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask in each task combination whose processing mode is batch processing includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost, and the query cost of the subtask in the processing mode of stream processing includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost; The target task combination is determined from the multiple task combinations according to the query cost of each task combination in the multiple task combinations.

7. A data query device, characterized in that: include: An acquisition module, used to acquire a query statement, wherein the query statement is used to issue a data query task; Processing modules for: Determine a plurality of task combinations, wherein each task combination includes a plurality of subtasks, the plurality of subtasks are obtained by splitting the data query task, and the processing mode of each subtask is a batch processing mode or a stream processing mode, wherein the plurality of task combinations include a first task combination and a second task combination, the first task combination and the second task combination include at least one different subtask, or the plurality of task combinations include a third task combination and a fourth task combination, the plurality of subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing modes of the plurality of subtasks in the third task combination and the combination of the processing modes of the plurality of subtasks in the fourth task combination are different; Based on the cost evaluation model, determining a target task combination from the multiple task combinations, wherein the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations; Perform data query according to the target task combination to obtain query results corresponding to the data query task.

8. The device according to claim 7, characterized in that The multiple subtasks are obtained by splitting the data query task according to at least one of the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task.

9. The device according to claim 7 or 8, characterized in that The processing method of each subtask is determined according to one or more of the following information: The operator type corresponding to the subtask, the structure of the query statement, and the writing status of the data source corresponding to the subtask.

10. The device according to any one of claims 7 to 9, characterized in that: The batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task; The processing module is used for: After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination; The target task combination is determined from the multiple task combinations according to the query cost corresponding to each task combination.

11. The device according to claim 10, characterized in that The processing module is used to execute the continuous query task or stream computing task after each subtask in the target task combination is converted to obtain the query result corresponding to the data query task.

12. The device according to any one of claims 7 to 11, characterized in that: The processing module is used for: Based on the cost evaluation model, determine the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask in each task combination whose processing mode is batch processing includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost, and the query cost of the subtask in the processing mode of stream processing includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost; The target task combination is determined from the multiple task combinations according to the query cost of each task combination in the multiple task combinations.

13. A computing device cluster, characterized in that: comprising at least one computing device, the at least one computing device comprising a processor and a memory; The processor is used to execute instructions stored in the memory so that the computing device cluster executes the method according to any one of claims 1-6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a processor, the processor is enabled to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 6.

Images