Data query method and related equipment

Abstract

The embodiment of the invention discloses a data query method, and the method comprises the steps: dividing a data query task into a plurality of sub-tasks, determining a feasible processing mode of each sub-task, and obtaining a plurality of task combinations according to the feasible processing mode of each sub-task. Wherein the processing modes of the sub-tasks in the task combination can influence the query cost when the data query task is executed based on the task combination, namely, the comprehensive cost of the execution efficiency and the execution cost of the data query task is often different under different combination modes of the multiple sub-tasks and the corresponding processing modes. Based on this, in the method, the target task combination with the lowest query cost can be determined from the plurality of task combinations through the cost evaluation model, so that the allocation of the batch tasks can be adaptively realized, and the resources of the batch-stream integrated data processing framework can be fully utilized to perform data query in a comprehensive optimal manner of query efficiency, cost and the like; and obtaining a query result.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 19, 2023, with Chinese application number 202310889683.X and invention name “A task processing method, device and computing device cluster”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the technical field of data query, and in particular to a data query method and related equipment. Background Art

[0003] As business scenarios continue to expand, in actual applications, we often face business requirements that require both stream data processing and batch data processing. To this end, data processing frameworks such as Spark and Flink currently support batch and stream integrated data processing.

[0004] However, in actual use, users need to manually allocate stream processing tasks and batch processing tasks required to complete data queries, so that the data processing framework can call stream processing resources to implement manually allocated stream processing tasks, and call batch processing resources to implement manually allocated batch processing tasks. This processing flow relies on manual experience to implement task allocation, which may not make good use of the resources of the data processing framework, resulting in the execution method of data query tasks not being the best method. Summary of the invention

[0005] The embodiment of the present application provides a data query method to solve the problem that the allocation of stream processing tasks and batch processing tasks may not be well utilized due to reliance on manual experience, resulting in the execution mode of the data query task not being a better mode. The present application also provides corresponding devices, equipment, computer-readable storage media, computer program products, etc.

[0006] The first aspect of the present application provides a data query method, which includes: obtaining a query statement, which is used to issue a data query task; determining multiple task combinations, wherein each task combination includes multiple subtasks, and the multiple subtasks are obtained by splitting the data query task, and the processing method of each subtask is a batch processing method or a stream processing method, wherein the multiple task combinations include a first task combination and a second task combination, and the first task combination and the second task combination include at least one different subtask, or the multiple task combinations include a third task combination and a fourth task combination, and the multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing methods of the multiple subtasks in the third task combination and the combination of the processing methods of the multiple subtasks in the fourth task combination are different; based on a cost evaluation model, determining a target task combination from the multiple task combinations, the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations; performing data query according to the target task combination to obtain a query result corresponding to the data query task.

[0007] In the first aspect, after splitting the data query task into multiple subtasks, the feasible processing method of each subtask can be determined, so as to obtain multiple task combinations according to the feasible processing method of each subtask. Among them, the processing method of the subtasks in the task combination will affect the query cost when executing the data query task based on the task combination, that is, under different combinations of multiple subtasks and corresponding processing methods, the execution efficiency and the comprehensive cost of the execution cost of the data query task are often different.

[0008] Based on this, in the first aspect, the cost evaluation model can be used to determine the target task combination with the lowest query cost from multiple task combinations, thereby adaptively realizing the allocation of stream and batch tasks, and making full use of the resources of the batch and stream integrated data processing framework to perform data queries in a comprehensive and optimal manner in terms of query efficiency and cost, and obtain query results.

[0009] In a possible implementation manner of the first aspect, the multiple subtasks are obtained by splitting the data query task according to at least one of a structure of a query statement, a query time range of the data query task, and aggregation time information of the data query task.

[0010] In this possible implementation, the data query task can be disassembled based on one or more splitting dimensions in the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task, so that the query load is broken up along one or more dimensions to reduce the granularity of the query load, improve data query efficiency, and facilitate the subsequent adaptive allocation of stream processing or batch processing to each subtask, thereby improving the overall query performance.

[0011] In a possible implementation manner of the first aspect, a processing manner of each subtask is determined according to one or more of the following information: an operator type corresponding to the subtask, a structure of a query statement, and a writing status of a data source corresponding to the subtask.

[0012] In this possible implementation, based on the characteristics and constraints of batch processing methods such as continuous query and stream processing methods such as stream computing, it can be determined that one or more of the operator type corresponding to the subtask, the structure of the query statement, and the write status of the data source corresponding to the subtask will have an impact on the feasible processing method of the subtask. Therefore, the feasible processing method of the subtask can be determined based on one or more of the above information.

[0013] In a possible implementation of the first aspect, the batch processing mode indicates that the subtasks are converted into continuous query tasks, and the stream processing mode indicates that the subtasks are converted into stream computing tasks; based on the cost evaluation model, a target task combination is determined from multiple task combinations, including: after each subtask in each task combination is converted into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination; according to the query cost corresponding to each task combination, the target task combination is determined from multiple task combinations.

[0014] In this possible implementation, continuous query is an automatically executed query that can periodically calculate and aggregate data and store the results in a new table. In the data query process, continuous query can reduce the query load and improve query efficiency by pre-calculating. Stream computing is a real-time data processing technology that can process and analyze data streams in real time to promptly discover and respond to events and trends in data streams. Stream computing is data-driven, that is, it calculates at write time, thereby reducing storage costs and improving query efficiency.

[0015] During the cost evaluation process, task conversion can be performed according to the processing method of subtasks in the task combination, and cost evaluation can be performed after task conversion to determine the target task combination with the lowest query cost, thereby realizing the adaptive allocation of stream batch tasks such as continuous query tasks and stream computing tasks, and performing data query in the most comprehensive way such as query efficiency and cost to obtain query results.

[0016] In a possible implementation of the first aspect, data query is performed according to the target task combination to obtain a query result corresponding to the data query task, including: executing a continuous query task or a stream computing task after each subtask in the target task combination is converted to obtain a query result corresponding to the data query task.

[0017] In a possible implementation of the first aspect, based on a cost evaluation model, a target task combination is determined from multiple task combinations, including: based on the cost evaluation model, determining the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask processed in batch mode in each task combination includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost, and the query cost of the subtask processed in stream processing mode includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost; according to the query cost of each task combination in the multiple task combinations, a target task combination is determined from the multiple task combinations.

[0018] In this possible implementation, the cost evaluation model can realize a comprehensive cost evaluation of the execution efficiency and execution cost (such as time cost and cost of consumed computing resources, etc.) of each task combination, thereby calculating the comprehensive execution cost of each task combination, so as to screen out the task combination with the best execution efficiency and execution cost from multiple task combinations as the target task combination.

[0019] Among them, the cost evaluation method of the subtask whose processing method is batch processing may be different from the cost evaluation method of the subtask whose processing method is stream processing. Specifically, since in the stream processing method, it is usually not necessary to write the data to a storage device such as a disk and then read it, but it is calculated when writing, so data scanning may not be performed, that is to say, the query cost of the subtask whose processing method is stream processing does not include the data scanning cost. Since there is no need to scan the data, there is no need to build a primary key index, etc. Therefore, the query cost of the subtask whose processing method is stream processing may include the data grouping cost, but the data grouping cost is different from the above-mentioned index scanning cost.

[0020] The second aspect of the present application provides a data query device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an acquisition module and a processing module.

[0021] A third aspect of the present application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, and the memory of at least one computing device stores computer execution instructions that can be run on the processor. When the computer execution instructions are executed by the processor, the processor executes the method as described in the first aspect or any possible implementation method of the first aspect.

[0022] A fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a computing device cluster, the computing device cluster executes a method as described in the first aspect or any possible implementation of the first aspect.

[0023] In a fifth aspect, the present application provides a computer program product storing one or more computer execution instructions. The computer program product includes computer execution instructions. When the computer execution instructions are executed by a computing device cluster, the computing device cluster executes a method as described in the first aspect or any possible implementation of the first aspect.

[0024] In a sixth aspect of the present application, a chip system is provided, which includes a processor for supporting a computing device cluster to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory, which is used to store necessary program instructions and data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0025] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of an embodiment of a data query method provided in an embodiment of the present application;

[0027] Figure 2 This is an exemplary schematic diagram of splitting a data query task to obtain multiple subtasks provided in an embodiment of the present application;

[0028] Figure 3a is an exemplary schematic diagram of a continuous query provided by an embodiment of the present application;

[0029] Figure 3b is an exemplary schematic diagram of stream computing provided by an embodiment of the present application;

[0030] Figure 4 is an exemplary schematic diagram of a processing method for determining a subtask provided in an embodiment of the present application;

[0031] Figure 5 is an exemplary schematic diagram of obtaining a target task combination provided in an embodiment of the present application;

[0032] Figure 6 is an exemplary structural diagram of a data query system provided in an embodiment of the present application;

[0033] Figure 7 is a schematic diagram of an embodiment of a data query device provided in an embodiment of the present application;

[0034] Figure 8 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0035] Fig. 9 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0036] Fig.10 It is another structural diagram of the computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0038] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0039] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a description of the way in which objects with the same attributes are described in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0040] Currently, data processing frameworks such as Spark and Flink support batch-stream integrated data query technology to meet various business needs such as stream data processing and batch data processing.

[0041] Batch-stream integrated data query technology is a technology that processes stream data and batch data in a unified manner. This technology can process both stream data in real-time scenarios and batch data in batch processing scenarios, thereby improving the efficiency and accuracy of data processing. This technology is widely used in many fields such as data warehouses, data analysis, and real-time monitoring.

[0042] In current actual use, when performing data query, users need to manually allocate the stream processing tasks and batch processing tasks required to complete the data query in data processing frameworks such as Spark and Flink. In other words, in current data processing frameworks, task processing methods are usually allocated based on manual experience, and stream processing tasks and batch processing tasks are obtained and executed through batch-stream integrated data query technology. Limited by manual experience, this task allocation method is difficult to fully utilize the resources in the data processing framework and determine the task execution method with better execution effect (for example, the execution cost is better after considering factors such as execution efficiency and execution cost).

[0043] Based on this, an embodiment of the present application provides a data query method that can solve the problem of relying on manual experience to implement the allocation of stream processing tasks and batch processing tasks, which may result in the inability to make good use of the resources of the data processing framework and determine the task execution method with better execution effect.

[0044] The method of the embodiment of the present application may be applied to a computing device cluster, which may include one or more computing devices.

[0045] The type of any computing device is not limited herein. Exemplarily, any computing device may be a terminal device, or a server, a container, or a virtual machine.

[0046] In the embodiment of the present application, the computing device cluster can realize batch-stream integrated data query, for example, a data processing framework such as Spark or Flink can be deployed in the computing device cluster. Alternatively, the computing device cluster can be connected to other devices that are deployed with data processing frameworks such as Spark or Flink to utilize the data processing frameworks in other devices to realize batch-stream integrated data query.

[0047] In one example, the computing device cluster can be used to implement a cloud platform. In this case, the embodiments of the present application can be applied to the cloud platform.

[0048] The cloud platform can provide computing, network and storage capabilities based on services of hardware resources and software resources. Exemplarily, the cloud platform can include one or more data centers to provide cloud resources through one or more data centers.

[0049] In an exemplary application scenario of a cloud platform, a cloud platform implemented by a computing device cluster can provide users with cloud services that can execute the methods of the embodiments of the present application, so that users can purchase or obtain the cloud services to implement data query.

[0050] Based on the computing device cluster, such as Figure 1 As shown, a data query method in an embodiment of the present application includes steps 101-104.

[0051] Step 101, obtaining a query statement.

[0052] Query statements are used to issue data query tasks.

[0053] The query statement may adopt Structured Query Language (SQL), or may adopt other existing or subsequently developed query languages.

[0054] The specific source of the query statement is not limited here.

[0055] In one example, the query statement is received by the computing device cluster through interaction with the user or a device deployed on the user side. For example, the device on the user side may send a query instruction carrying the query statement to the computing device cluster, so that the computing device cluster obtains the query statement and triggers the execution of the corresponding data query task. Alternatively, the user may input the query statement to the computing device cluster so that the computing device cluster obtains the query statement and triggers the execution of the corresponding data query task.

[0056] In another example, in a big data scenario, the computing device cluster may be a computing device cluster including multiple computing nodes. During the process of analyzing and processing data, the computing device cluster generates the query statement and triggers the execution of the corresponding data query task.

[0057] The data query task refers to obtaining query results from a specified data source according to a query statement. The data source may be located in a computing device cluster executing the embodiment of the present application, or may be stored in other devices such as a database.

[0058] The specific type of the data query task is not limited here. Exemplarily, the data query task may be to perform a data query on stream data and / or batch data.

[0059] Step 102: determine multiple task combinations.

[0060] Each task combination includes multiple subtasks, and the multiple subtasks are obtained by splitting the data query task. The processing method of each subtask is batch processing or stream processing. The multiple task combinations include a first task combination and a second task combination, and the first task combination and the second task combination include at least one different subtask, or the multiple task combinations include a third task combination and a fourth task combination, and the multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing methods of the multiple subtasks in the third task combination and the combination of the processing methods of the multiple subtasks in the fourth task combination are different.

[0061] In the embodiment of the present application, the process of determining multiple task combinations may include the following processing stages:

[0062] Split the data query task, determine the feasible processing method for each subtask, and combine the subtasks to obtain multiple task combinations.

[0063] The following is an exemplary introduction to each processing stage in the process.

[0064] A. Split data query tasks.

[0065] There are many specific ways to split the data query task, which are not limited here.

[0066] In some examples, the multiple subtasks are obtained by splitting the data query task according to at least one of the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task.

[0067] In this way, the data query task can be disassembled based on one or more splitting dimensions in the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task, so that the query load is broken up along one or more dimensions to reduce the granularity of the query load, improve data query efficiency, and facilitate the subsequent adaptive allocation of stream processing or batch processing to each subtask, thereby improving the overall query performance.

[0068] Below, several possible splitting dimensions and corresponding splitting methods are introduced by way of example.

[0069] 1. The structure of the query statement.

[0070] In some examples, the data query task may be split based on the splitting dimension of the structure of the query statement.

[0071] For example, by parsing the structure of the query statement, it can be determined that the query statement includes three levels of nested subqueries. Then, according to the query level, the data query task can be split into three subtasks on the splitting dimension of the query statement structure.

[0072] 2. The time range of the data query task.

[0073] The time range of a data query task may include the time period to be queried by the data query task. For example, if the query statement indicates to query the specified data within 3 days from January 1, 2023 to January 3, 2023, then the time range of the data query task is 3 days. In this example, based on the splitting dimension of the time range of the data query task, the data query task can be split according to a smaller time granularity or time interval, and the data query task can be split within the time range. For example, the day is split to obtain 3 subtasks, each of which corresponds to a day within the 3-day time period.

[0074] 3. Aggregation time information of data query tasks.

[0075] The aggregation time information of the data query task may include the duration of the aggregation time window of the data query task. Generally speaking, the aggregation time window of the data query task may be a time window for performing aggregation operations within the time range of the data query task. For example, the query instruction indicates to query the specified data within 3 days from January 1, 2023 to January 3, 2023, and the aggregation time information of the data query task indicates that the aggregation operation is performed within the 3-day period with an aggregation time window of 1 hour. Then, in this example, the aggregation time information of the data query task may be 1 hour. In this example, based on the splitting dimension of the aggregation time information of the data query task, the data query task may be split into multiple subtasks within the preset 1-hour aggregation time window according to a smaller time granularity, such as 1 minute, 10 minutes, 30 minutes, etc. It can be understood that the aggregation time of the data query task is a finer-grained time window within the time range of the data query task.

[0076] It can be seen that by splitting the data query task based on time information such as the query time range of the data query task and the aggregation time information of the data query task, the query load of the current data query task can be dispersed along the time dimension. In the time dimension, the granularity of the query load is reduced, thereby reducing the query load within a unit time range. In this way, under the condition of limited computing resources, query operations such as high-load and big data queries can be efficiently executed, thereby reducing the resource cost and time cost of data query and improving data query efficiency.

[0077] For example, Figure 2 The figure shows an exemplary schematic diagram of splitting a data query task along multiple splitting dimensions.

[0078] After obtaining the query instruction carrying the query statement, the data query task is split on three splitting dimensions: the structure of the query statement, the time range of the data query task, and the aggregation time of the data query task.

[0079] Among them, in terms of the structure of the query statement, it is determined that the query statement SQL includes three layers of nested subqueries, namely level1, level2 and level3. Then, in terms of the structure of the query statement, the data query task is split into three subtasks, namely SQL (level1), SQL (level2) and SQL (level3).

[0080] In the time range corresponding to the data query task, since the query instruction indicates to query the specified data within the three days of day1, day2 and day3, the data query task can be split into three subtasks SQL (day1), SQL (day2), and SQL (day3) within this time range.

[0081] In terms of the aggregation time corresponding to the data query task, the aggregation time can be 1 hour, 2 hours, and 3 hours. The data query task can be split into three subtasks, SQL (1 hour), SQL (2 hours), and SQL (3 hours), within this time range.

[0082] B. Determine feasible handling methods for each subtask.

[0083] After splitting the data query task into multiple subtasks, the possible processing methods for each subtask can be determined.

[0084] Each subtask may be processed in batch mode and / or stream mode.

[0085] That is to say, for a subtask, it can be determined, for example, through conversion rules, whether the subtask can be executed in batch processing, or in stream processing, or both in batch processing and in stream processing.

[0086] When a subtask is processed in a batch processing mode, the subtask can be processed using batch processing resources and technologies (such as batch processing resources and technologies in a data processing engine).

[0087] The specific content of the batch processing method is not limited here. The batch processing method may be an existing batch processing method or a batch processing method developed subsequently.

[0088] For example, a batch processing method may be continuous query (CQ).

[0089] A continuous query is an automatically executed query that can periodically calculate and aggregate data and store the results in a new table. In the data query process, continuous queries can reduce the query load and improve query efficiency by pre-calculating.

[0090] like Figure 3a In the example shown, before the continuous query, the data source is usually pre-stored in the underlying storage device. When defining a continuous query, you need to specify the time window size and the forward sliding times. During the continuous query process, after each time window is closed and the next time window begins, the data in the just closed time window is calculated (for example, performing scheduled calculations, aggregating data, etc.), and the calculation results are saved, for example, stored in the storage device. Therefore, the overall process of continuous query is to query data from the bottom up, and then write the corresponding calculation results from the top down. The query load can be reduced and the query efficiency can be improved by calculating the data of each time window in advance.

[0091] In some other examples, the batch processing method may also include other processing methods besides continuous query, for example, it may include one or more processing methods such as pre-aggregation and down-sampling.

[0092] When the processing mode of the subtask is stream processing, the subtask can be processed by using stream processing resources and technologies (such as stream processing resources and technologies in a data processing engine).

[0093] The specific content of the stream processing method is not limited here. The stream processing method may be an existing stream processing method or a subsequently developed stream processing method.

[0094] For example, one stream processing method may be stream computing.

[0095] Stream computing is a real-time data processing technology that can process and analyze data streams in real time to promptly discover and respond to events and trends in the data stream. Stream computing is data-driven, that is, it calculates at write time, thereby reducing storage costs and improving query efficiency.

[0096] like Figure 3bIn the example shown, an exemplary data transmission process of stream computing can be that after receiving the real-time data stream, the stream computing results are obtained through information processing and interaction such as filtering, local aggregation (LocalAgg), global aggregation (GlobaAgg) and stream management modules, and storage is achieved through output operations such as flush, for example, the stream computing results are stored in the underlying disk and other storage devices. It can be seen that the overall process of stream computing is top-down, and calculations are performed while writing data.

[0097] In the embodiment of the present application, optional batch processing methods (such as continuous query, pre-aggregation, downsampling, etc.) and optional stream processing methods (such as stream computing, etc.) can be predetermined.

[0098] After obtaining multiple subtasks, it can be determined based on the conversion rules whether each subtask is suitable for batch processing or stream processing, thereby determining a feasible processing method for each subtask.

[0099] In addition, in some examples, there may be more than one batch processing mode, and / or there may be more than one stream processing mode. For example, in the conversion rules, the optional batch processing modes include batch processing mode 1 and batch processing mode 2, and the optional stream processing mode includes stream processing mode 1. For subtask B, if it is determined according to the conversion rules that subtask B is suitable for both batch processing mode 1 and batch processing mode 2 and stream processing mode 1, then subtask B1, subtask B2, and subtask B3 of subtask B can be obtained. Among them, the processing mode corresponding to subtask B1 is batch processing mode 1, the processing mode corresponding to subtask B2 is batch processing mode 2, and the processing mode of subtask B2 is stream processing mode 1.

[0100] In the embodiment of the present application, the specific content of the conversion rule for determining the feasible processing mode of the subtask is not limited here. Exemplarily, the conversion rule can be pre-configured by the developer according to the characteristics and constraints of the batch processing mode and the stream processing mode.

[0101] The following uses batch processing as continuous query and stream processing as stream computing as examples to illustrate the characteristics and constraints of continuous query and stream computing.

[0102] Continuous query can have the following characteristics: 1. Support high-order operators and complex queries; 2. Support distributed scheduling and load balancing.

[0103] Continuous queries may have the following constraints: 1. During the continuous query process, the amount of data is large, which may lead to large disk input / output (I / O) and network overhead; 2. If multiple continuous query tasks are to execute queries on the same table, each continuous query task must read the table separately, resulting in repeated reading of data in the table, which may cause timing conflicts, delays, and waste of resource overhead.

[0104] Stream computing can have the following characteristics: 1. Calculation is performed through memory, which is fast and has little impact on storage; 2. High computing efficiency, low disk I / O and network overhead.

[0105] Stream computing can have the following constraints: 1. The cache will reside in the memory, which may increase the memory overhead; 2. Only incremental computing operators are supported, such as sum / count / minimum / max operators.

[0106] In this way, based on the characteristics and constraints of batch processing methods such as continuous query and stream processing methods such as stream computing, transformation rules can be configured to determine feasible processing methods for subtasks through the transformation rules.

[0107] In some embodiments, the processing manner of each subtask is determined according to one or more of the following information:

[0108] The operator type corresponding to the subtask, the structure of the query statement, and the writing status of the data source corresponding to the subtask.

[0109] An exemplary introduction is given below.

[0110] 1. Operator type corresponding to the subtask.

[0111] In some examples, a feasible processing method for a subtask can be determined according to the operator type corresponding to the subtask. For example, if the operator type corresponding to the subtask can be executed by batch processing, then one feasible processing method for the subtask is batch processing. If the operator type corresponding to the subtask can also be executed by stream processing, then another feasible processing method for the subtask is stream processing.

[0112] For example, the operator corresponding to a subtask is an aggregation operator. Since the aggregation operator does not support incremental calculation, it cannot be implemented through stream processing, but can only be implemented through batch processing such as continuous query. In this case, the subtask only corresponds to batch processing, which is a feasible processing method.

[0113] 2. The structure of the query statement.

[0114] In different query scenarios, the structure of the query statement may be different. Therefore, the relationship between the subtasks obtained by splitting the data query task on the splitting dimension of the query statement structure may be different. For example, in some scenarios, the subtasks obtained by splitting the data query task based on the structure of the query statement may be multi-layer nested relationships, or parallel and independent relationships. Since the relationship between subtasks affects the feasible processing methods of the subtasks, the feasible processing methods of the subtasks can be determined based on the structure of the query statement.

[0115] For example, if the data query task is split according to the structure of the query statement to obtain three subtasks, and according to the structure of the query statement, it can be determined that among the three subtasks, the input data of subtask B is the output data of subtask A, and subtask B can be driven by the output data of subtask A, then this subtask B can adopt stream processing methods such as stream computing.

[0116] 3. The writing status of the data source corresponding to the subtask.

[0117] The data source corresponding to the subtask refers to the data object to be processed or queried by the subtask. In different processing scenarios, the storage location of the data source may be different.

[0118] For example, in some scenarios, the write status of the data source corresponding to a subtask is that the data source has been pre-stored in a storage device such as a hard disk in the computing device cluster or a storage device external to the computing device cluster. At this time, based on the write status, the subtask can be processed by batch processing such as continuous query.

[0119] In other scenarios, the write status of the data source corresponding to a subtask is that the data source has not been written to a storage device such as a hard disk. In this case, the data source can be a real-time input data stream in a real-time scenario, that is, the data source is streaming data. In this case, the data source can be stored in the memory. Since the data source is not written to a storage device such as a hard disk, the corresponding subtask can be processed using stream processing methods such as stream computing.

[0120] After splitting the data query task into multiple subtasks and determining the possible processing methods for each subtask, multiple task combinations can be obtained by combining the multiple subtasks and the possible processing methods for each subtask. Each task combination can include at least two subtasks, and the processing method of each subtask is batch processing or stream processing.

[0121] For example, Figure 4 The figure is an exemplary schematic diagram for determining a feasible processing method for each subtask.

[0122] exist Figure 4 In the example shown, based on the structure of the query statement, it is determined that the query statement SQL includes three levels of nested subqueries, namely level1, level2 and level2. Then, based on the structure of the query statement, the data query task is split into three subtasks, namely SQL1 (level1), SQL1 (level2) and SQL1 (level3).

[0123] According to the conversion rules, it is determined that the operator corresponding to the subtask SQL1 (level1) cannot use stream processing methods such as batch computing, but can only use stream processing. Therefore, the feasible processing method for the subtask SQL1 (level1) is stream processing. In addition, according to the conversion rules, it is determined that the operator corresponding to the subtask SQL (level2) can only use batch processing and cannot use stream processing. Therefore, the feasible processing method for the subtask SQL (level2) is batch processing. According to the conversion rules, it is determined that the operator corresponding to the subtask SQL (level3) can use both stream processing and batch processing. Therefore, the feasible processing methods for the subtask SQL (level3) are batch processing and stream processing.

[0124] C. Combine subtasks to obtain multiple task combinations.

[0125] After determining the feasible processing methods for each subtask, multiple task combinations can be obtained.

[0126] Each task combination includes multiple subtasks, and the multiple subtasks are obtained by splitting the data query task. The processing method of each subtask is batch processing or stream processing. The multiple task combinations include a first task combination and a second task combination, and the first task combination and the second task combination include at least one different subtask, or the multiple task combinations include a third task combination and a fourth task combination, and the multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing methods of the multiple subtasks in the third task combination and the combination of the processing methods of the multiple subtasks in the fourth task combination are different.

[0127] In the embodiment of the present application, each task combination can complete the data query task.

[0128] There are differences between different task combinations. Specifically, there may be differences in subtasks and / or there may be the same subtasks but the same subtasks are processed in different ways.

[0129] The following example illustrates this.

[0130] In one example, the plurality of task combinations include a first task combination and a second task combination, and subtasks of the first task combination and subtasks of the second task combination may be different. In addition, in some examples, when the first task combination and the second task combination include the same subtasks, the processing methods of the same subtasks may be the same or different.

[0131] For example, in one example, the first task combination and the second task combination are split in different ways.

[0132] For example, when the structure of a query statement is nested, the data query task is split only according to the structure of the query statement to obtain three subtasks, namely subtask A, subtask B and subtask C; based on this splitting method, a first task combination is obtained, and the first task combination includes subtask A, subtask B and subtask C.

[0133] In addition, the data query task can also be split according to the structure of the query statement. After obtaining subtasks A, subtask B and subtask C, subtask B can be split into subtask B1 and subtask B2 according to the time range of the data query task, and subtask C can be split into subtask C1 and subtask C2. Then, the second task combination can be obtained according to the splitting method. Specifically, the second task combination includes 5 subtasks, namely subtask A, subtask B1, subtask B2, subtask C1 and subtask C2.

[0134] It can be seen that, in this example, the first task combination and the second task combination include at least one different subtask.

[0135] In addition, among the multiple task combinations, there may be any two task combinations that have the same subtasks, but the same subtasks are processed in different ways.

[0136] Specifically, the multiple task combinations include a third task combination and a fourth task combination. The multiple subtasks in the third task combination and the fourth task combination are the same, but the combination of processing methods of the multiple subtasks in the third task combination and the combination of processing methods of the multiple subtasks in the fourth task combination are different.

[0137] For example, when the structure of the query statement is nested, the data query task is split only according to the structure of the query statement to obtain three subtasks, namely subtask A, subtask B and subtask C.

[0138] Then, it can be determined that the feasible processing modes of subtask A are batch processing and stream processing, the feasible processing modes of subtask B are batch processing and stream processing, and the feasible processing mode of subtask C is batch processing.

[0139] In this way, four task combinations can be obtained in this example. Multiple subtasks in the four task combinations are the same, but there are differences in the combinations of processing methods of the corresponding multiple subtasks.

[0140] For example, the four task combinations may include a third task combination, which includes subtask A, subtask B and subtask C, and the processing method of subtask A in the third task combination is batch processing, the processing method of subtask B is batch processing, and the processing method of subtask C is batch processing.

[0141] Furthermore, the four task combinations may include a fourth task combination, which includes subtask A, subtask B and subtask C. In addition, the processing method of subtask A in the fourth task combination is stream processing, the processing method of subtask B is batch processing, and the processing method of subtask C is batch processing.

[0142] It can be seen that, in this example, the processing method of subtask A in the third task combination is different from the processing method of subtask A in the fourth task combination.

[0143] Step 103 : Based on the cost evaluation model, a target task combination is determined from the multiple task combinations, and the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations.

[0144] In some examples, a cost evaluation model may be used to perform cost evaluation on each of the multiple task combinations to obtain a task combination with the lowest query cost as a target task combination.

[0145] In other examples, when there are a large number of task combinations, these large number of task combinations can be preliminarily screened through evolutionary algorithms (such as genetic algorithms or particle swarm algorithms) to delete task combinations with obviously higher query costs, and then the target task combination can be determined from the preliminarily screened task combinations through a cost evaluation model.

[0146] This cost evaluation model can realize a comprehensive cost evaluation of the execution efficiency and execution cost (such as time cost and the cost of consumed computing resources, etc.) of each task combination, thereby calculating the comprehensive execution cost of each task combination, and screening out the task combination with the best execution efficiency and execution cost from multiple task combinations as the target task combination.

[0147] The specific content of the cost evaluation model can be various. For example, the cost evaluation method of a subtask in a batch processing mode may be different from the cost evaluation method of a subtask in a stream processing mode. Specifically, the cost evaluation can be performed according to the processing mode of the subtask.

[0148] In some embodiments, during the cost evaluation process, task conversion may be performed according to the processing method of subtasks in the task combination, and cost evaluation may be performed after the task conversion.

[0149] Specifically, in some embodiments, the batch processing mode indicates converting the subtask into a continuous query task, and the stream processing mode indicates converting the subtask into a stream computing task;

[0150] Based on the cost evaluation model, the target task combination is determined from multiple task combinations, including:

[0151] After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination;

[0152] According to the query cost corresponding to each task combination, a target task combination is determined from multiple task combinations.

[0153] In an embodiment of the present application, after each subtask in each task combination is converted into a continuous query task or a stream computing task, the query cost corresponding to each task combination can be determined based on the cost evaluation method of the continuous query task and the cost evaluation method of the stream processing task.

[0154] Below, in the cost evaluation model, the cost evaluation method of the subtask whose processing method is a batch processing method such as continuous query and the cost evaluation method of the subtask whose processing method is a stream processing method such as stream computing can be exemplarily introduced respectively.

[0155] 1. Cost evaluation method for subtasks in batch processing mode.

[0156] In one embodiment, the query cost of the subtask processed in batch mode includes at least one of the following cost evaluation indicators:

[0157] Index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost.

[0158] The following is an explanation of each cost assessment indicator with specific examples.

[0159] In an exemplary scenario A, a subtask that adopts a batch processing method adopts a primary key index method.

[0160] The timeline of this subtask is M, involving n indexes. The cardinality (number of different elements) of the i-th index is card(indexs[i]), the cardinality of the aggregate field is I, and the cardinality of the primary key is P.

[0161] The number of rows of data to be queried is N, and the average number of rows for each primary key is N / P.

[0162] The number of data files is F, and the number of files to be queried is Fq, among which the number of ordered files is Fq1, the number of unordered files is Fq2, and the file selection rate is Fq / F.

[0163] The number of ordered rows for each primary key is (N / P) / Fq*Fq1, and the number of out-of-order rows for each primary key is (N / P) / Fq*Fq2.

[0164] In this scenario, each file uses k hash functions to generate a bloom filter. The data layout in each file is divided into different chunks according to the timeline. Each chunk has corresponding metadata (called chunkMeta), and every 1024 chunkMeta corresponds to a metadata index (metaIndex).

[0165] The ratio of the amount of data after query filtering to the total amount of data is called the selectivity. In this scenario, if multi-timeline spatiotemporal aggregation query is implemented based on technologies such as open Gemini, and no filtering is done on the timeline dimension, the selectivity is M / M, almost 100%. If the number of background timelines is M and the number of timelines selected by the query is S, the corresponding selectivity is S / M. The amount of query data is N. The number of spatial and temporal groups is Gs and Gt respectively, and the total number of groups G = Gs*Gt.

[0166] In the cost evaluation model, the algorithm complexity corresponding to one or more cost evaluation indicators can be considered. For example, in the scenario of multi-timeline spatiotemporal aggregation query, multiple aspects such as indexing, data scanning, data calculation, data conversion, and data transmission all involve overhead in terms of algorithm complexity. Among them, the complexity coefficients corresponding to these multiple aspects can be different, but they are uniformly represented by the C (correlation coefficient) correlation coefficient.

[0167] In this scenario, the specific calculation method of the cost evaluation index corresponding to the subtask in batch processing mode is as follows:

[0168] 1) Index scan cost.

[0169] The index scan cost can reflect the resource cost of the index scan operation corresponding to the corresponding subtask, such as time cost and computing resource overhead, etc. The index scan cost may be related to the index mode, related index algorithm, etc.

[0170] For example, in scenario A above, the index scan cost is calculated by the following terms:

[0171]

[0172] Among them, considering that the primary key groups will be traversed during index scanning and the order between groups and within groups is maintained, the index scanning cost is related to the primary key cardinality P and the complexity of group calculation.

[0173] Specifically, in this scenario, the grouping computation complexity is calculated by to describe.

[0174] 2) Data scanning cost.

[0175] The overhead of scanning data is reflected in filtering files through Bloom filters, finding the corresponding chunks in each file through binary search, and scanning data in each chunk.

[0176] In the above scenario A, the data scanning cost is calculated by the following items:

[0177]

[0178] The following explains the meaning of each of the above items:

[0179] P×F q ×k: Determine whether each primary key is in the file;

[0180] Determine whether the data block of each primary key is in the file;

[0181] Scan data from sorted and unsorted files.

[0182] 3) Data computation cost.

[0183] The data computation cost may include, for example, aggregation computation cost.

[0184] For the aggregation calculation cost, the main cost of spatial aggregation is the number of rows of data to be queried N. Exemplarily, in a system acceleration query calculation technology, considering the vectorization calculation coefficient v of vectorization, the data calculation cost can be calculated by the following items:

[0185]

[0186] Among them, C is the coefficient corresponding to the cost of calculating the data.

[0187] In some examples, the upper-layer client and other devices have requirements for the data form, such as requiring order, so the data form needs to be converted, for example, the data needs to be sorted, which will bring corresponding overhead. Based on this, the cost evaluation index can include the data conversion cost, specifically, the data conversion cost can include the sorting cost, etc.

[0188] Among them, for the sorting cost, the main sorting overhead may be the sorting of calculation results among multiple groups.

[0189] For example, in the above scenario A, the calculation result of each group is calculated as 1 value, and the data conversion cost can be calculated by the following items:

[0190]

[0191] Among them, v is the vectorization calculation coefficient of vectorization, and C is the coefficient corresponding to the data conversion cost.

[0192] 5) Data transmission cost.

[0193] For data transfer cost, the data transfer cost may come from the data after aggregation, for example, the cost of reading data from the bottom up.

[0194] In the above scenario A, suppose that the spatiotemporal aggregation is based on n fields (for example, based on the region or other attributes of the hosts in the server cluster) and is grouped according to a1, a2, ..., an, and the grouping cardinality is card (a i ), the data transmission cost can be calculated by the following terms:

[0195]

[0196] Wherein, C is the coefficient corresponding to the data transmission cost.

[0197] 6) Data writing cost.

[0198] Cost of data writing write The main data writing overhead may include the overhead of writing data into storage space such as a hard disk, memory, etc. and / or the overhead of the CPU storing data from the storage space.

[0199] 7) Interface interaction cost.

[0200] Cost of interface interaction IO ,Interface interaction overhead can come from interface interaction operations such as memory ,interface, disk interface, etc.

[0201] Thus, in the above scenario A, based on the above costs, the cost of the subtask in batch processing is CQ It can be calculated based on the following formula:

[0202]

[0203] Of course, in some other examples, the cost evaluation index of the subtask in batch processing may include one or more of the above examples, and the specific calculation method of the above cost evaluation index may be different based on different scenario requirements. For example, when the index method adopts a common index, the specific calculation method of the index scan cost may be different from the above example.

[0204] 2. The query cost of the subtask whose processing mode is stream processing mode includes at least one of the following cost evaluation indicators:

[0205] Data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost.

[0206] Among them, since in the stream processing method, it is usually not necessary to write data to a storage device such as a disk and then read it, but calculation is performed while writing, so data scanning can be omitted, that is, the query cost of the subtask whose processing method is the stream processing method does not include the data scanning cost. Since there is no need to scan data, there is no need to build a primary key index, etc. Therefore, the query cost of the subtask whose processing method is the stream processing method can include the data grouping cost, but this data grouping cost is different from the above-mentioned index scanning cost.

[0207] Therefore, illustratively, in the above scenario A, referring to the above example of the subtask using batch processing, the cost value of the subtask using stream processing is SC It can be calculated based on the following formula:

[0208]

[0209] The data packet cost is

[0210] In this way, under the condition of certain computing resources, the optimal combination of batch processing subtasks and stream processing subtasks can be determined through cost evaluation, and the target task combination can be obtained, so as to rationally utilize resources to achieve efficient data query and improve query performance.

[0211] For example, in Figure 5 In the example shown, the data query task is divided into three subtasks based on the structure of the query statement, namely subtask SQL (level 1), subtask SQL (level 2) and subtask SQL (level 3). The three subtasks are three-level nested subqueries.

[0212] Taking the three subtasks in the query statement structure as an example, possible forms of target task combinations are introduced.

[0213] And in Figure 5In the example shown, based on the cost evaluation model, it can be seen that the subtask SQL (level 1) has better processing performance when the stream processing method is used. Therefore, in the target task combination, the subtask SQL (level 1) is processed by the stream processing method.

[0214] In addition, the processing performance of subtask SQL (level2) is better when it is processed in batch mode. Therefore, in the target task combination, subtask SQL (level2) is processed in batch mode. In this way, the processing of flow batches is adaptively realized from subtask SQL (level1) to subtask SQL (level2).

[0215] Due to the nested relationship, the processing performance of the process of using the calculation data writing of subtask SQL (level2) as a driver to implement the calculation flow of subtask SQL (level3) is better. In the target task combination, subtask SQL (level3) is processed in a stream processing manner, and the calculation data writing of subtask SQL (level2) in batch processing can be used as the input stream data of subtask SQL (level3).

[0216] It can be seen that in this example, the appropriate processing method for each subtask in the target task combination can be adaptively determined according to the cost evaluation model, and in this example, the batch-stream conversion can be realized in the target task combination determined according to the cost evaluation model, thereby making full use of the resources of the batch-stream integrated data query system to execute data query tasks, so as to improve data query efficiency.

[0217] Step 104: perform data query according to the target task combination to obtain query results corresponding to the data query task.

[0218] In some examples, the batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task. Then, the continuous query task or stream computing task after each subtask in the target task combination is executed to obtain the query result corresponding to the data query task.

[0219] In an embodiment of the present application, after determining the target task combination, a directed acyclic graph (DAG) can be used to construct an execution plan and perform task scheduling and execution based on the continuous query task and / or stream computing task in the target task combination, so as to obtain the query result corresponding to the data query task.

[0220] In the embodiment of the present application, after splitting the data query task, the feasible processing method of each subtask can be adaptively determined, so as to obtain multiple task combinations according to the feasible processing method of each subtask. Among them, the processing method of the subtask in the task combination will affect the query cost when executing the data query task based on the task combination, that is, under different combinations of multiple subtasks and corresponding processing methods, the execution efficiency and the comprehensive cost of the execution cost of the data query task are often different.

[0221] Based on this, in the embodiments of the present application, the target task combination with the lowest query cost can be determined from multiple task combinations through a cost evaluation model, thereby adaptively realizing the allocation of stream and batch tasks, and making full use of the resources of the batch-stream integrated data processing framework to perform data queries in a comprehensive and optimal manner in terms of query efficiency and cost, and obtain query results.

[0222] Any of the above data query method embodiments may be implemented by one or more functional modules in a data query system deployed in a computing device cluster.

[0223] In combination with any of the above embodiments, an exemplary introduction is given below to a data query system that can be deployed in the computing device cluster.

[0224] For example, Figure 6 The figure shows an exemplary framework diagram of a data query system implemented based on the computing device cluster.

[0225] The data query system may include Figure 6 One or more of the multiple functional modules shown are used to implement any of the above-mentioned data query method embodiments.

[0226] The following is an exemplary introduction to the functional modules that may be included in the data query system.

[0227] In the data query system, interfaces such as query statements such as structured query language (SQL), data streams and / or machine learning (ML) application program interfaces (API) can be provided as entries for query instructions carrying query statements, data streams, and data analysis information based on machine learning.

[0228] In addition, the data query system may include one or more of a parser, a task identification module, a task decomposition module, a rule module, a cost module, a DAG construction module, a DAG scheduling module, and a DAG execution module.

[0229] Specifically, the parser can parse the query statement, etc., so as to identify the content of the data query task. For example, the query statement can be parsed to obtain a query syntax tree, and then a query object is generated through syntax and semantic analysis to obtain a parsing result. Then, the parsing result can be transmitted to the task identification module to identify the type of the data query task and identify the type of the data query task. The specific type of the data query task is not specifically limited here.

[0230] After the data query task is identified, the data query task can be decomposed through a task decomposition module to obtain a corresponding plurality of subtasks.

[0231] Then, the rule module can be used to determine the feasible processing methods of the subtasks according to the conversion rules, and obtain multiple task combinations. Then, the cost module can be used to obtain the cost value of each task combination according to the cost evaluation model, so as to determine the optimal cost combination from multiple task combinations as the target task combination.

[0232] After obtaining the target task combination, an execution plan can be constructed through the DAG construction module, and then the task scheduling can be implemented through the DAG scheduling module and executed through one or more executors in the DAG execution module, so as to obtain the query result corresponding to the data query task.

[0233] The data query system may be applied in a variety of scenarios, that is, any of the above-mentioned data query method embodiments of the present application may be applied in a variety of scenarios, which are not limited here.

[0234] In some examples, the data query system can be used for database query, and can also be used for scenarios such as big data management, etc. Moreover, the specific functions of the data query system can be determined based on the scenario.

[0235] For example, the method of decomposing a data query task to obtain multiple subtasks in any of the above embodiments can be applied to a rule-based optimizer (RBO) in a data query system to decompose queries with large data volumes along one or more splitting dimensions such as the time dimension, thereby dispersing large query loads along the time dimension and reducing overall computing resource overhead.

[0236] In addition, the method for performing task combination evaluation based on the cost evaluation model in any of the above embodiments can be applied to a cost-based query optimizer (CBO) in a data query system to perceive the query load and adaptively adjust stream batch computing tasks to improve query efficiency.

[0237] The data query method provided in the embodiment of the present application is described above from multiple aspects. The data query device provided in the embodiment of the present application is described below in conjunction with the accompanying drawings.

[0238] like Figure 7 As shown, the embodiment of the present application provides a data query device 70, and the data query device 70 includes:

[0239] The acquisition module 701 is used to acquire a query statement, and the query statement is used to issue a data query task;

[0240] The processing module 702 is used to:

[0241] Determine a plurality of task combinations, wherein each task combination includes a plurality of subtasks, the plurality of subtasks are obtained by splitting a data query task, and each subtask is processed in a batch processing manner or a stream processing manner, wherein the plurality of task combinations include a first task combination and a second task combination, the first task combination and the second task combination include at least one different subtask, or the plurality of task combinations include a third task combination and a fourth task combination, the plurality of subtasks in the third task combination and the fourth task combination are the same, but a combination of processing methods for the plurality of subtasks in the third task combination and a combination of processing methods for the plurality of subtasks in the fourth task combination are different;

[0242] Based on the cost evaluation model, a target task combination is determined from multiple task combinations, and the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations;

[0243] Perform data query according to the target task combination to obtain the query results corresponding to the data query task.

[0244] Optionally, the multiple subtasks are obtained by splitting the data query task according to at least one of the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task.

[0245] Optionally, the processing mode of each subtask is determined according to one or more of the following information:

[0246] The operator type corresponding to the subtask, the structure of the query statement, and the writing status of the data source corresponding to the subtask.

[0247] Optionally, the batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task;

[0248] The processing module 702 is used for:

[0249] After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination;

[0250] According to the query cost corresponding to each task combination, a target task combination is determined from multiple task combinations.

[0251] Optionally, the processing module 702 is used to: execute the continuous query task or stream computing task after each subtask in the target task combination is converted to obtain a query result corresponding to the data query task.

[0252] Optionally, the processing module 702 is used to:

[0253] Based on the cost evaluation model, determine the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask in each task combination whose processing mode is batch processing includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost, and the query cost of the subtask in the processing mode of stream processing includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost;

[0254] According to the query cost of each task combination in the multiple task combinations, a target task combination is determined from the multiple task combinations.

[0255] In an embodiment of the present application, a module is used as an example of a software functional unit, and the data query device may include code running on a computing instance. Among them, the computing instance may be at least one of computing devices such as a physical host (computing device), a virtual machine, and a container. Furthermore, the above-mentioned computing device may be one or more. For example, the data query device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same available zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0256] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0257] As an example of a hardware functional unit, the data query device may include at least one computing device, such as a server, etc. Alternatively, the data query device may also be a device implemented using an application specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex PLD (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0258] The multiple computing devices included in the data query device can be distributed in the same region or in different regions. The multiple computing devices included in the data query device can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data query device can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0259] It should be noted that, in other embodiments, the acquisition module can be used to execute any step in the data query method, and the processing module can be used to execute any step in the data query method. The steps that the acquisition module and the processing module are responsible for implementing can be specified as needed. The full functions of the data query device can be realized by respectively implementing different steps in the data query method through the acquisition module and the processing module.

[0260] The present application embodiment also provides a computing device 80. Figure 8 As shown, the computing device 80 includes: a bus 82, a processor 84, a memory 86 and a communication interface 88. The processor 84, the memory 86 and the communication interface 88 communicate with each other through the bus 82. The computing device 80 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 80.

[0261] The bus 82 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 84 is shown as a single line, but does not mean that there is only one bus or one type of bus. The bus 84 may include a path for transmitting information between various components of the computing device 80 (eg, the memory 86, the processor 84, the communication interface 88).

[0262] The processor 84 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0263] The memory 86 may include a volatile memory, such as a random access memory (RAM). The processor 84 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0264] The memory 86 stores executable program codes, and the processor 84 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module and training module, thereby implementing the data query method applied to the computing device cluster in the above-mentioned embodiment, that is, the memory 86 stores instructions for executing the data query method applied to the computing device cluster in the above-mentioned embodiment.

[0265] The communication interface 88 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 80 and other devices or communication networks.

[0266] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0267] like Fig. 9 As shown, the computing device cluster includes at least one computing device 80. The memory 86 in one or more computing devices 80 in the computing device cluster may store the same instructions for executing the data query method.

[0268] In some possible implementations, the memory 86 of one or more computing devices 80 in the computing device cluster may also store partial instructions for executing the data query method. In other words, the combination of one or more computing devices 80 may jointly execute instructions for executing the data query method.

[0269] It should be noted that the memory 86 in different computing devices 80 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data query method. That is, the instructions stored in the memory 86 in different computing devices 80 can implement the functions of one or more modules in the acquisition module and the training module.

[0270] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 80A and 80B are connected via a network. Specifically, they are connected to the network via the communication interface in each computing device. In this type of possible implementation, the memory 86 in the computing device 80A may store instructions for executing the functions of the acquisition module. At the same time, the memory 86 in the computing device 80B may store instructions for executing the functions of the processing module. Alternatively, the memory 86 in the computing device 80A may store instructions for executing part of the functions of the processing module. At the same time, the memory 86 in the computing device 80B may store instructions for executing another part of the functions of the processing module.

[0271] It should be understood that Fig.10 The functions of the computing device 80A shown in FIG. 8A may also be completed by multiple computing devices 80. Similarly, the functions of the computing device 80B may also be completed by multiple computing devices 80.

[0272] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.10 The difference is that the memory 86 in one or more computing devices 80 in the computing device cluster may store the same instructions for executing the data query method.

[0273] In some possible implementations, the memory 86 of one or more computing devices 80 in the computing device cluster may also store partial instructions for executing the data query method. In other words, the combination of one or more computing devices 80 may jointly execute instructions for executing the data query method.

[0274] It should be noted that the memory 86 in different computing devices 80 in the computing device cluster can store different instructions for executing part of the functions of the data query method. That is, the instructions stored in the memory 86 in different computing devices 80 can implement the functions of one or more modules in the acquisition module and the training module.

[0275] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data query method.

[0276] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data query method.

[0277] The embodiment of the present application also provides a chip system, which includes a processor, and the processor is used to implement the steps performed by the above-mentioned computing device cluster. In a possible design, the chip system may also include a memory, and the memory is used to store necessary program instructions and data. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0278] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0279] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0280] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0281] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0282] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A data query method, characterized in that: The method comprises: Obtaining a query statement, wherein the query statement is used to issue a data query task; Determine a plurality of task combinations, wherein each task combination includes a plurality of subtasks, the plurality of subtasks are obtained by splitting the data query task, and the processing mode of each subtask is a batch processing mode or a stream processing mode, wherein the plurality of task combinations include a first task combination and a second task combination, the first task combination and the second task combination include at least one different subtask, or the plurality of task combinations include a third task combination and a fourth task combination, the plurality of subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing modes of the plurality of subtasks in the third task combination and the combination of the processing modes of the plurality of subtasks in the fourth task combination are different; Based on the cost evaluation model, determining a target task combination from the multiple task combinations, wherein the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations; Perform data query according to the target task combination to obtain query results corresponding to the data query task.

2. The method according to claim 1, characterized in that The multiple subtasks are obtained by splitting the data query task according to at least one of the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task.

3. The method according to claim 1 or 2, characterized in that: The processing method of each subtask is determined according to one or more of the following information: The operator type corresponding to the subtask, the structure of the query statement, and the writing status of the data source corresponding to the subtask.

4. The method according to any one of claims 1 to 3, characterized in that The batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task; The step of determining a target task combination from the plurality of task combinations based on the cost evaluation model includes: After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination; The target task combination is determined from the multiple task combinations according to the query cost corresponding to each task combination.

5. The method according to claim 4, characterized in that The performing of data query according to the target task combination to obtain a query result corresponding to the data query task includes: Execute the continuous query task or stream computing task converted from each subtask in the target task combination to obtain the query result corresponding to the data query task.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining a target task combination from the plurality of task combinations based on the cost evaluation model includes: Based on the cost evaluation model, determine the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask in each task combination whose processing mode is batch processing includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost, and the query cost of the subtask in the processing mode of stream processing includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost; The target task combination is determined from the multiple task combinations according to the query cost of each task combination in the multiple task combinations.

7. A data query device, characterized in that: include: An acquisition module, used to acquire a query statement, wherein the query statement is used to issue a data query task; Processing modules for: Determine a plurality of task combinations, wherein each task combination includes a plurality of subtasks, the plurality of subtasks are obtained by splitting the data query task, and the processing mode of each subtask is a batch processing mode or a stream processing mode, wherein the plurality of task combinations include a first task combination and a second task combination, the first task combination and the second task combination include at least one different subtask, or the plurality of task combinations include a third task combination and a fourth task combination, the plurality of subtasks in the third task combination and the fourth task combination are the same, but the combination of the processing modes of the plurality of subtasks in the third task combination and the combination of the processing modes of the plurality of subtasks in the fourth task combination are different; Based on the cost evaluation model, determining a target task combination from the multiple task combinations, wherein the query cost of the target task combination is lower than that of other task combinations in the multiple task combinations; Perform data query according to the target task combination to obtain query results corresponding to the data query task.

8. The device according to claim 7, characterized in that The multiple subtasks are obtained by splitting the data query task according to at least one of the structure of the query statement, the query time range of the data query task, and the aggregation time information of the data query task.

9. The device according to claim 7 or 8, characterized in that The processing method of each subtask is determined according to one or more of the following information: The operator type corresponding to the subtask, the structure of the query statement, and the writing status of the data source corresponding to the subtask.

10. The device according to any one of claims 7 to 9, characterized in that: The batch processing mode indicates that the subtask is converted into a continuous query task, and the stream processing mode indicates that the subtask is converted into a stream computing task; The processing module is used for: After converting each subtask in each task combination into a continuous query task or a stream computing task, the cost evaluation model is input to determine the query cost corresponding to each task combination; The target task combination is determined from the multiple task combinations according to the query cost corresponding to each task combination.

11. The device according to claim 10, characterized in that The processing module is used to execute the continuous query task or stream computing task after each subtask in the target task combination is converted to obtain the query result corresponding to the data query task.

12. The device according to any one of claims 7 to 11, characterized in that: The processing module is used for: Based on the cost evaluation model, determine the query cost of each task combination in the multiple task combinations, wherein the query cost of the subtask in each task combination whose processing mode is batch processing includes at least one of the following: index scanning cost, data scanning cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost, and the query cost of the subtask in the processing mode of stream processing includes at least one of the following: data grouping cost, data calculation cost, data transmission cost, data writing cost, data conversion cost, and interface interaction cost; The target task combination is determined from the multiple task combinations according to the query cost of each task combination in the multiple task combinations.

13. A computing device cluster, characterized in that: comprising at least one computing device, the at least one computing device comprising a processor and a memory; The processor is used to execute instructions stored in the memory so that the computing device cluster executes the method according to any one of claims 1-6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a processor, the processor is enabled to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 6.

Images