Method and system for scheduled processing of pre-aggregation computations for time-series databases

Abstract

The application discloses a planning processing method and system for pre-aggregation calculation of a time series database, belongs to the technical field of time series databases, and aims to solve the technical problem of how to quickly query the aggregation calculation results of each collection point. The method comprises the following steps: obtaining a pre-aggregation calculation statement input by a user; calling a logical execution plan generation interface of a parser to analyze the pre-aggregation calculation statement, analyzing the pre-aggregation calculation statement into a corresponding logical plan, and analyzing the result type of the aggregation rule of the pre-aggregation calculation statement; creating a result table according to the obtained result type, wherein the result table is used for storing the aggregation calculation results; calling an execution plan generation interface of the parser to generate a general execution plan based on the logical plan; clipping the execution plan to obtain a clipped execution plan; and matching the clipped execution plan with the result table and executing the result table.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of time series database, in particular to a plan processing method and system for time series database pre-aggregation calculation. BACKGROUND

[0002] Time series data is mainly generated by various types of real-time monitoring, inspection and analysis equipment, and the time series data has the characteristics of fast generation frequency, large data volume and strong dependence on time. The pre-aggregation calculation is to set a query rule in advance, group the time series data in real time through a time bucket function, and aggregate the time series data in real time according to the rule. The aggregated results are stored in a result table, and users can quickly obtain the aggregation results by querying the result table. This is the whole process of pre-aggregation calculation.

[0003] How to quickly query the aggregation calculation results of each collection point is a technical problem to be solved. SUMMARY

[0004] The technical task of the present application is to provide a plan processing method and system for time series database pre-aggregation calculation to solve the technical problem of how to quickly query the aggregation calculation results of each collection point.

[0005] In a first aspect, the present application provides a plan processing method for time series database pre-aggregation calculation, comprising the following steps:

[0006] obtaining a pre-aggregation calculation statement input by a user;

[0007] calling a logical execution plan generation interface of a parser to parse the pre-aggregation calculation statement, parsing the pre-aggregation calculation statement into a corresponding logical plan, and parsing the result type of the aggregation rule of the pre-aggregation calculation statement, creating a result table according to the obtained result type, wherein the result table is used to store the aggregation calculation results;

[0008] calling an execution plan generation interface of the parser to generate a general execution plan based on the logical plan;

[0009] trimming the execution plan to obtain a trimmed execution plan;

[0010] matching the trimmed execution plan with the result table and executing the trimmed execution plan.

[0011] As a preferred, before obtaining the pre-aggregation calculation statement input by the user, the following operations are performed:

[0012] generating a pre-aggregation calculation query statement through syntax analysis, and marking a flag for the pre-aggregation calculation statement;

[0013] For the user input query statement, a logical plan generation interface is called to judge whether the query statement is a pre-aggregation calculation statement based on a flag, and if so, the pre-aggregation calculation statement is parsed.

[0014] As preferred, the result type of the pre-aggregation calculation statement is parsed to obtain a structure type including an aggregation rule aggregation column ID and an aggregation type, the aggregation rule aggregation column ID and the aggregation type are stored in the logical plan, and a result table is created according to the aggregation rule aggregation column ID and the aggregation type.

[0015] As preferred, the structure body of the execution plan includes three parts, namely render, column id and column type;

[0016] When the execution plan is trimmed, the following operations are performed: time_bucket in render is deleted, if the column processed by time_bucket is not involved in other render items, the column type and column id processed by time_bucket are deleted in column type and column id, otherwise, only timebucket in renders is deleted, and outputcols and outputtypes are not deleted.

[0017] In a second aspect, the present application discloses a plan processing system for pre-aggregation calculation of a time series database, which is characterized in that the system is used for executing the plan processing method for pre-aggregation calculation of the time series database according to any one of the first aspect, and the system comprises:

[0018] A statement acquisition module is configured to acquire a pre-aggregation calculation statement input by a user.

[0019] A statement parsing module is configured to call a logical execution plan generation interface of a parser to parse the pre-aggregation calculation statement, parse the result type of an aggregation rule of the pre-aggregation calculation statement, and create a result table according to the obtained result type, wherein the result table is used to store an aggregation calculation result.

[0020] An execution plan generation module is configured to call an execution plan generation interface of the parser to generate a general execution plan based on the logical plan.

[0021] An execution plan trimming module is configured to trim the execution plan to obtain a trimmed execution plan.

[0022] A matching execution module is configured to match the trimmed execution plan with the result table and execute the trimmed execution plan.

[0023] As preferred, the system further comprises a statement configuration module, configured to generate a pre-aggregation computing query statement through syntax analysis, and mark a flag for the pre-aggregation computing query statement.

[0024] The statement analysis module is further configured to: for a query statement input by a user, call a logical plan generation interface, judge whether the query statement is a pre-aggregation computing query statement based on the flag, and if yes, analyze the pre-aggregation computing query statement.

[0025] As preferred, the statement analysis module is configured to analyze a result type of the query statement aggregation rule, obtain a structure type comprising an aggregation rule aggregation column ID and an aggregation type, store the aggregation rule aggregation column ID and the aggregation type into a logical plan, and create a result table according to the aggregation rule aggregation column ID and the aggregation type.

[0026] As preferred, the structure body of the execution plan comprises three parts, namely, render, column id and column type.

[0027] When the execution plan is pruned, the execution plan pruning module is configured to: delete time_bucket in render, if the column processed by time_bucket does not involve other render items, delete the column type and the column id of the column processed by time_bucket in column type and column id, otherwise, only delete timebucket in renders, and do not delete outputcols and outputtypes.

[0028] The plan processing method and system for pre-aggregation computing of the time series database have the following advantages: a general query execution plan generation interface can be called to generate a special execution plan for pre-aggregation computing, elements required for pre-aggregation computing can be saved with a small storage cost, and a simple expression structure can be quickly matched by an execution plan and a result table column, so that the construction of pre-aggregation computing can be quickly and effectively completed, and the query of the user's aggregation computing is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] The present application will be further described below with reference to the drawings.

[0031] Figure 1 A flow chart of a plan processing method for pre-aggregation calculation of a time series database according to an embodiment 1. DETAILED DESCRIPTION

[0032] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it. The embodiments and technical features in the embodiments can be combined with each other without conflict, if possible.

[0033] The embodiments of the present application provide a plan processing method and system for pre-aggregation calculation of a time series database, which are used to solve the technical problem of how to quickly query the aggregation calculation results of each collection point.

[0034] Embodiment 1

[0035] As shown in the figure, the plan processing method for pre-aggregation calculation of a time series database according to the present application comprises the following steps: Figure 1

[0036] S100, obtaining a pre-aggregation calculation statement input by a user;

[0037] S200, calling a logical execution plan generation interface of a parser to parse the pre-aggregation calculation statement, parsing the pre-aggregation calculation statement into a corresponding logical plan, and parsing a result type of an aggregation rule of the pre-aggregation calculation statement, creating a result table according to the obtained result type, the result table being used to store aggregation calculation results;

[0038] S300, calling an execution plan generation interface of the parser to generate a general execution plan based on the logical plan;

[0039] S400, clipping the execution plan to obtain a clipped execution plan;

[0040] S500, matching the clipped execution plan with the result table and executing the same.

[0041] In the embodiment, before obtaining the pre-aggregation calculation statement input by the user, the following operation is performed: generating a pre-aggregation calculation query statement through syntax parsing, and marking a flag for the pre-aggregation calculation statement.

[0042] Correspondingly, when the step S200 is performed, for the query statement input by the user, the logical plan generation interface is called to judge whether the query statement is a pre-aggregation calculation statement based on the flag, and if so, the pre-aggregation calculation statement is parsed.

[0043] ​In step S200, the result type of the pre-aggregation calculation statement is parsed to obtain a structure type including an aggregation rule aggregation column ID and an aggregation type, the aggregation rule aggregation column ID and the aggregation type are stored into a logical plan, and a result table is created according to the aggregation rule aggregation column ID and the aggregation type.

[0044] The structure body of the execution plan obtained in step S300 includes three parts, namely, render, column id and column type.

[0045] When the execution plan is trimmed in step S400, the following operations are performed: time_bucket in render is deleted, if the column processed by time_bucket does not involve other render items, the column type and the column id processed by time_bucket are deleted in column type and column id, otherwise, only timebucket in renders is deleted, and outputcols and outputtypes are not deleted.

[0046] The result table has a time sequence when created, and the time_bucket function in the pre-aggregation calculation statement affects the matching of the aggregation rule column and the result table column, and cannot be executed correctly, so the time_bucket function related in the pre-aggregation calculation statement execution plan is deleted.

[0047] In the execution process of the embodiment, the pre-aggregation calculation is created first by syntax analysis to generate a query statement of the pre-aggregation calculation, and then a logical plan generation interface is called to parse the query statement to generate a corresponding logical plan. Unlike a general query statement, the query statement of the pre-aggregation calculation parses the result type of the query statement aggregation rule when the logical plan generation interface is called, as shown in the following formula: Figure 1 The result type includes an aggregation rule aggregation column id and an aggregation column type, and then a result table is created through the result type, and the result table is used to store the result of the aggregation calculation.

[0048] After the result table is successfully created, the pre-calculation calls an execution plan generation interface, takes the logical plan of the pre-calculation as a parameter, generates a corresponding execution plan, and the structure body of the execution plan mainly includes three parts, namely, render, column id and column type. First, time_bucket in render is deleted, if the column processed by time_bucket does not involve other render items, then the column type and the column id processed by time_bucket are deleted in column type and column id, otherwise, only timebucket in renders is deleted, and outputcols and outputtypes are not deleted. Finally, the trimmed execution plan is encoded and sent to an execution engine for matching and corresponding processing.

[0049] Embodiment 2

[0050] The application discloses a planning processing system for pre-aggregation calculation of a time-series database, which comprises a statement acquisition module, a statement analysis module, an execution plan generation module, an execution plan clipping module and a matching execution module.

[0051] The statement acquisition module is used for acquiring a pre-aggregation calculation statement input by a user.

[0052] The statement analysis module is used for calling a logical execution plan generation interface of a parser to analyze the pre-aggregation calculation statement, analyzing the pre-aggregation calculation statement into a corresponding logical plan, and analyzing a result type of an aggregation rule of the pre-aggregation calculation statement, creating a result table according to the obtained result type, wherein the result table is used for storing an aggregation calculation result.

[0053] In the embodiment, the statement analysis module is used for analyzing the result type of the pre-aggregation calculation statement, obtaining an aggregation rule aggregation column ID and an aggregation type, storing the aggregation rule aggregation column ID and the aggregation type into the logical plan, and creating a result table according to the aggregation rule aggregation column ID and the aggregation type.

[0054] The execution plan generation module is used for calling an execution plan generation interface of the parser to generate a general execution plan based on the logical plan.

[0055] In the embodiment, the structure of the execution plan comprises three parts, namely, render, column ID and column type.

[0056] The execution plan clipping module is used for clipping the execution plan to obtain a clipped execution plan.

[0057] When the execution plan is clipped, the execution plan clipping module in the embodiment is used for performing the following operations: deleting time_bucket in render, if the column processed by time_bucket does not involve other render items, deleting the column type and the column ID of the column processed by time_bucket in the column type and the column ID, otherwise, only deleting timebucket in renders, and not deleting outputcols and outputtypes.

[0058] The result table has a time sequence when created, and the time_bucket function in the pre-aggregation calculation statement affects the matching between the aggregation rule column and the result table column, and cannot be correctly executed, so the time_bucket function related in the execution plan of the pre-aggregation calculation statement is deleted.

[0059] The matching execution module is configured to match the clipped execution plan with the result table and to issue the execution. In the specific implementation process, the matching execution module is configured to encode the clipped execution plan and send it to the execution engine for matching and corresponding processing.

[0060] As an improvement of the present embodiment, the system further comprises a statement configuration module, which is configured to generate a pre-aggregation computing query statement through syntax analysis and mark a flag for the pre-aggregation computing query statement.

[0061] Correspondingly, the statement analysis module is further configured to perform the following: for a query statement input by a user, calling a logical plan generation interface, judging whether the query statement is a pre-aggregation computing query statement based on the flag, and if so, analyzing the pre-aggregation computing query statement.

[0062] The present application has been described in detail through the accompanying drawings and preferred embodiments, however, the present application is not limited to these disclosed embodiments, and based on the above-mentioned embodiments, those skilled in the art can know that the means in the above-mentioned different embodiments can be combined to obtain more embodiments of the present application, and these embodiments are also within the protection scope of the present application.

Claims

1. A method for planning a processing of a temporal database pre-aggregation computation, characterized in that, The method comprises the following steps: obtaining a pre-aggregation computing statement input by a user; calling a logical execution plan generation interface of a parser to parse the pre-aggregation computing statement, parsing the pre-aggregation computing statement into a corresponding logical plan, and parsing a result type of an aggregation rule of the pre-aggregation computing statement, creating a result table according to the obtained result type, wherein the result table is used to store an aggregation calculation result, wherein the structure type obtained by parsing the result type of the pre-aggregation computing statement comprises an aggregation rule aggregation column ID and an aggregation type, the aggregation rule aggregation column ID and the aggregation type are stored in the logical plan, and a result table is created according to the aggregation rule aggregation column ID and the aggregation type; calling an execution plan generation interface of the parser to generate a general execution plan based on the logical plan; trimming the execution plan to obtain a trimmed execution plan, wherein the structure body of the execution plan comprises three parts, namely, render, column id and column type; when the execution plan is trimmed, the following operations are performed: deleting time_bucket in render, if the column processed by time_bucket does not involve other render items, deleting the column type and column id of the column processed by time_bucket in the column type and column id, otherwise, only deleting timebucket in renders, and not deleting outputcols and outputtypes; matching the trimmed execution plan with the result table and delivering execution.

2. The method of claim 1, wherein the plan processing method is a pre-aggregation computation of a time series database. Before obtaining the pre-aggregation computing statement input by the user, the following operations are performed: generating a pre-aggregation computing query statement through syntax parsing, and marking a flag for the pre-aggregation computing statement; for the query statement input by the user, calling a logical plan generation interface to determine whether the query statement is a pre-aggregation computing statement based on the flag, and if so, parsing the pre-aggregation computing statement.

3. A system for plan processing of a temporal database pre-aggregation computation, characterized in that, A system for performing the plan processing method of the time series database pre-aggregation calculation according to any one of claims 1-2, the system comprising: a statement obtaining module, configured to obtain a pre-aggregation computing statement input by a user; a statement parsing module, configured to call a logical execution plan generation interface of a parser to parse the pre-aggregation computing statement, parse the pre-aggregation computing statement into a corresponding logical plan, and parse a result type of an aggregation rule of the pre-aggregation computing statement, create a result table according to the obtained result type, wherein the result table is used to store an aggregation calculation result, wherein the structure type obtained by parsing the result type of the pre-aggregation computing statement comprises an aggregation rule aggregation column ID and an aggregation type, the aggregation rule aggregation column ID and the aggregation type are stored in the logical plan, and a result table is created according to the aggregation rule aggregation column ID and the aggregation type; an execution plan generation module, configured to call an execution plan generation interface of the parser to generate a general execution plan based on the logical plan; The execution plan pruning module is configured to prune the execution plan to obtain a pruned execution plan, and a structure of the execution plan includes three parts, namely, render, column id, and column type; when the execution plan is pruned, the following operations are performed: time_bucket in render is deleted, if the column processed by time_bucket is not involved in other render items, the column type and column id processed by time_bucket are deleted in the column type and column id, otherwise, outputcols and outputtypes are not deleted, and only timebucket in renders is deleted; The matching execution module is configured to match the pruned execution plan with a result table and issue an execution.

4. The system of claim 3, wherein, The system further includes a statement configuration module configured to generate a pre-aggregation calculation query statement through syntax analysis and mark a flag for the pre-aggregation calculation query statement; The statement analysis module is further configured to perform the following: for a query statement input by a user, a logical plan generation interface is called, whether the query statement is a pre-aggregation calculation query statement is judged based on the flag, if yes, the pre-aggregation calculation query statement is analyzed.

Images