A method, device, equipment and storage medium for data aggregation at multiple time granularities

Abstract

The present application discloses a data aggregation method, apparatus, device and storage medium with multiple time granularities, which relates to the field of data processing, including: configuring multiple data sources to obtain several databases, and constructing table names for each data table in the database based on a preset time granularity; configuring task information through a task configuration interface to obtain a to-be-executed aggregation task; judging whether to trigger the task execution operation step by step according to the time granularity, and if so, determining the aggregation time range based on the time granularity in the task information, and finding the source data table and the destination data table from the source database and the destination database; the source database and the destination database are databases determined from several databases; and saving the to-be-aggregated data determined from the source data table based on the aggregation time range to the destination data table. The present application generates corresponding to-be-executed aggregation tasks based on the time granularity and aggregates them step by step according to the time granularity, which can effectively reduce the amount of aggregated data and improve aggregation efficiency and flexibility.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, equipment and storage medium for data aggregation at multiple time granularities. Background Art

[0002] Cloud computing systems generate a large amount of performance data, which may reside in different databases and data tables. Existing technologies for data aggregation have the following shortcomings: First, adding a new data table requires redeveloping the aggregation module, which is inefficient. Second, data aggregation for any time range is always performed from the original data table, resulting in large amounts of data being aggregated each time, slowing down processing. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a data aggregation method, device, equipment, and storage medium with multiple time granularities, which can effectively reduce the amount of aggregated data and improve aggregation efficiency and flexibility. The specific scheme is as follows:

[0004] In a first aspect, the present application provides a data aggregation method at multiple time granularities, comprising:

[0005] Configuring multiple data sources to obtain multiple databases, and constructing table names corresponding to the corresponding time granularities for each data table in the database based on preset time granularities;

[0006] Configure task information through the task configuration interface to obtain the corresponding summary tasks to be executed;

[0007] Determine whether a corresponding task execution operation is currently triggered according to the time granularity, and if so, determine a corresponding summary time range based on the time granularity recorded in the task information, and search the source data table and the destination data table pre-recorded in the task information from the source database and the destination database; the source database and the destination database are databases determined from the plurality of databases;

[0008] Based on the summary time range, the corresponding data to be summarized is determined from the source data table and the data to be summarized is saved to the target data table to complete the summary operation of the data to be summarized.

[0009] Optionally, determining corresponding data to be aggregated from the source data table based on the aggregation time range and saving the data to be aggregated to the destination data table includes:

[0010] Based on the summary time range and the source field pre-recorded in the task information, corresponding data to be summarized is determined from the source data table, and the data to be summarized is saved in the destination data table.

[0011] Optionally, the multi-time granularity data aggregation method further includes:

[0012] Utilize the aggregation time range and the aggregation algorithm pre-recorded in the task information and the source field to obtain the corresponding data to be aggregated;

[0013] Execute the first data judgment operation pre-recorded in the task information to determine whether the primary key of the data to be summarized is repeated in the target data table, and if so, perform corresponding data update;

[0014] Executing a second data determination operation pre-recorded in the task information to determine whether the data to be aggregated is a timestamp, and if so, performing a query within a corresponding time range;

[0015] executing a third data determination operation pre-recorded in the task information to determine whether the data to be aggregated is a device ID, and if so, performing a corresponding data grouping operation;

[0016] The data to be summarized is saved in the destination data table using the destination field pre-recorded in the task information.

[0017] Optionally, the stepwise determination of whether a corresponding task execution operation is currently triggered according to the time granularity includes:

[0018] Start the preset scheduled task to regularly determine whether the corresponding task execution operation is triggered at the time granularity starting from 0:00 of each hour.

[0019] Optionally, after completing the aggregation operation on the data to be aggregated, the method further includes:

[0020] Based on the table name of the source data table corresponding to the summary task to be executed, the table name of the destination data table, the corresponding time granularity and the summary time range, a corresponding task execution record is determined and the record is saved in a preset task execution information table.

[0021] Optionally, the stepwise determination of whether a corresponding task execution operation is currently triggered according to the time granularity includes:

[0022] Determine whether the current time is 0 minutes and 0 seconds. If so, execute the corresponding hourly granularity pending summary task;

[0023] When the execution of the summary task to be executed at the hourly granularity is completed, judging whether there is a corresponding record in the preset task execution information table based on the first summary start time corresponding to the summary task to be executed at the daily granularity, if not, executing the summary task to be executed at the daily granularity;

[0024] When the execution of the summary task to be executed at the daily granularity is completed, judging whether there is a corresponding record in the preset task execution information table based on the second summary start time corresponding to the summary task to be executed at the weekly granularity, and if not, executing the summary task to be executed at the weekly granularity;

[0025] When the weekly summary task to be executed is completed, judging whether there is a corresponding record in the preset task execution information table based on the third summary start time corresponding to the monthly summary task to be executed, and if not, executing the monthly summary task to be executed;

[0026] When the execution of the to-be-executed summary task of the monthly granularity is completed, judging whether there is a corresponding record in the preset task execution information table based on the fourth summary start time corresponding to the to-be-executed summary task of the quarterly granularity, and if not, executing the to-be-executed summary task of the quarterly granularity;

[0027] When the execution of the summary task to be executed at the quarterly granularity is completed, it is determined whether there is a corresponding record in the preset task execution information table based on the fifth summary start time corresponding to the summary task to be executed at the annual granularity. If not, the summary task to be executed at the annual granularity is executed.

[0028] Optionally, constructing a table name corresponding to the corresponding time granularity for each data table in the database based on the preset time granularity includes:

[0029] A corresponding table name suffix is ​​constructed based on a preset time granularity, and the table name suffix is ​​added to the table name of each data table in the current database to obtain a table name corresponding to the corresponding time granularity; wherein the table name suffix corresponds to the time granularity one by one.

[0030] In a second aspect, the present application provides a data aggregation device with multiple time granularities, comprising:

[0031] Database configuration module, used to configure multiple data sources to obtain several databases;

[0032] A table name construction module, configured to construct a table name corresponding to a corresponding time granularity for each data table in the database based on a preset time granularity;

[0033] The task configuration module is used to configure task information through the task configuration interface to obtain the corresponding summary tasks to be executed;

[0034] a task execution module, configured to determine, step by step according to the time granularity, whether a corresponding task execution operation is currently triggered; if so, to determine a corresponding summary time range based on the time granularity recorded in the task information, and to search a source data table and a destination data table pre-recorded in the task information from a source database and a destination database; the source database and the destination database being databases determined from the plurality of databases;

[0035] The data aggregation module is used to determine the corresponding data to be aggregated from the source data table based on the aggregation time range and save the data to be aggregated to the destination data table to complete the aggregation operation on the data to be aggregated.

[0036] In a third aspect, the present application provides an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] The processor is used to execute the computer program to implement the steps of the aforementioned multi-time granularity data aggregation method.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of the aforementioned multi-time granularity data aggregation method.

[0040] It can be seen that in this application, first, multiple data sources are configured to obtain several databases, and table names corresponding to the corresponding time granularity are constructed for each data table in the database based on the preset time granularity; then, task information is configured through the task configuration interface to obtain the corresponding summary task to be executed; then, whether the corresponding task execution operation is currently triggered is determined step by step according to the time granularity, and if so, the corresponding summary time range is determined based on the time granularity recorded in the task information, and the source data table and the destination data table pre-recorded in the task information are found from the source database and the destination database; the source database and the destination database are databases determined from the several databases; based on the summary time range, the corresponding data to be summarized are determined from the source data table and the data to be summarized are saved to the destination data table to complete the summary operation of the data to be summarized. This application generates the corresponding summary task to be executed based on the time granularity and summarizes it step by step according to the time granularity, which can effectively reduce the repeated development caused by the addition of new data tables when summarizing data in the existing technology, reduce the amount of summarized data, and improve the summary efficiency and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A flow chart of a multi-time granularity data aggregation method provided in this application;

[0043] Figure 2 A flowchart of a specific multi-time granularity data aggregation method provided in this application;

[0044] Figure 3 A flow chart of a specific multi-time granularity data aggregation method provided in this application;

[0045] Figure 4 This is a schematic diagram of the structure of a multi-time granularity data aggregation device provided by this application;

[0046] Figure 5 This is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] In cloud computing systems, a large amount of performance data is generated, and this performance data may exist in different databases and different data tables. The existing technology has the following shortcomings when performing data aggregation: First, if a new data table is added, the aggregation module must be redeveloped, which is inefficient. Second, regardless of the time range, the data aggregation is all aggregated from the original data table, and the amount of data aggregated each time is large, which affects the processing speed. To this end, the present application provides a data aggregation solution with multiple time granularities, which can effectively reduce the repeated development of data aggregation, reduce the amount of aggregated data, and improve aggregation efficiency and flexibility.

[0049] See also Figure 1 As shown, an embodiment of the present invention discloses a data aggregation method with multiple time granularities, including:

[0050] Step S11: configure multiple data sources to obtain several databases, and construct table names corresponding to corresponding time granularities for each data table in the database based on preset time granularities.

[0051] In one specific implementation, this embodiment supports InfluxDB (a time series database designed to handle high write and query loads) and MySQL (a relational database management system, also a relational database), both of which support multiple data sources. It should be understood that constructing table names corresponding to the corresponding time granularity for each data table in the database based on the preset time granularity includes: constructing corresponding table name suffixes based on the preset time granularity, and adding the table name suffixes to the table names of each data table currently in the database to obtain table names corresponding to the corresponding time granularity; wherein the table name suffixes correspond one-to-one with the time granularity. The preset time granularity includes hour, day, week, month, quarter, and year. Accordingly, table name suffixes constructed sequentially for the preset time granularity may include _hour, _day, _week, _month, _quarter, and _year. The original data table does not have a time suffix. Based on the above table name rules, it can be ensured that the summarized data comes from the data table of the next level of time granularity.

[0052] Step S12: Configure task information through the task configuration interface to obtain corresponding summary tasks to be executed.

[0053] In this embodiment, after configuring the database and constructing table names corresponding to the corresponding time granularity for each data table in the database based on the preset time granularity, it is necessary to configure task information through the task configuration interface to obtain the corresponding summary task to be executed.

[0054] Specifically, first, select a database from the configured databases as the source database, denoted as srcDatabase. Then, select a database from the configured database source as the destination database, denoted as dstDatabase. The source database and the destination database can be the same or different databases. Then, after determining the source database and the destination database, automatically query the data tables in the source database and the destination database. For the InfluxDB database, the following query statement can be used:

[0055] show measurements;

[0056] For the MySQL database, you can use the following statement to query:

[0057] SELECT*FROM information_schema.TABLES WHERE TABLE_SCHEMA='#database';

[0058] You can also use the following statement to query:

[0059] SELECT*FROM information_schema.TABLES WHERE TABLE_SCHEMA='#database'AND table_name LIKE'%_day';

[0060] When querying, replace #database with the actual database name. %_day represents a query for table names ending in _day, specifically, the daily data table. Furthermore, the source data table can be denoted as srcTable, and the destination data table as dstTable. Specifically, the rules for filtering data tables based on time granularity are shown in Table 1 below.

[0061] Table 1

[0062]

[0063]

[0064] Where xx is the table name without the time suffix. If the selected time granularity is daily, tables with the _hour suffix in their names will be filtered out from the source data table, and tables with the _day suffix in their names will be filtered out from the destination data table. If the selected time granularity is hourly, tables without the above suffixes will be filtered out from the source data table, and tables with the _hour suffix in their names will be filtered out from the destination data table.

[0065] Furthermore, after selecting the source data table and the target data table, all fields in the table will be automatically queried. For the data table in the influxdb database, the following statement can be used to query:

[0066] show tag keys from table_name;

[0067] show field keys from table_name;

[0068] For the data tables in the MySQL database, you can use the following statement to query:

[0069] SELECT*FROM information_schema.COLUMNS WHERE TABLE_SCHEMA='#database'AND TABLE_NAME='#table_name'

[0070] When actually querying, it is necessary to replace #database with the real database name, and replace #table_name with the real table name of the source data table or the destination data table. Among them, the field that needs data aggregation is selected from the field list of the source data table, which is called the source field and is recorded as srcField. Afterwards, a series of settings including clustering algorithms and corresponding judgments can be made for the source field, and a field is selected from the field list of the destination data table, which is called the destination field and is recorded as dstField. It can be understood that when data aggregation is performed, the value corresponding to the source field is calculated and saved to the destination field. After the configuration is completed, the above configuration will be persisted to the corresponding database, and this configuration will be used as the corresponding summary task to be executed.

[0071] Step S13: Determine whether the corresponding task execution operation is currently triggered according to the time granularity. If so, determine the corresponding summary time range based on the time granularity recorded in the task information, and find the source data table and destination data table pre-recorded in the task information from the source database and the destination database; the source database and the destination database are databases determined from the several databases.

[0072] In this embodiment, according to the time granularity, it is necessary to first complete the summary tasks to be executed at the hourly granularity, and then execute the summary tasks to be executed at the daily granularity, weekly granularity, monthly granularity, quarterly granularity, and annual granularity in sequence, so as to ensure the integrity of the data when summarizing the data. Figure 2As shown, first start the preset scheduled task to regularly judge whether the corresponding task execution operation is triggered according to the time granularity starting from 0:00 every hour. As long as the preset scheduled task is started normally, the corresponding hourly granularity summary task to be executed can be executed, and then the corresponding summary time range is determined based on the current start time of the preset scheduled task, and the summary time range is one hour before the current start time. After that, based on the summary time range, query whether there is a corresponding summary task to be executed. If not, determine the first summary start time corresponding to the current daily granularity summary task to be executed based on the start time, and determine whether the task execution operation corresponding to the daily granularity summary task to be executed is triggered by judging whether there is a corresponding record in the preset task execution information table. If so, obtain the corresponding task information to perform the corresponding data aggregation, and after the hourly granularity summary task to be executed is executed, judge whether the task execution operation corresponding to the daily granularity summary task to be executed is triggered based on the first summary start time. If not, the daily summary task to be executed is executed. If yes, a second summary start time corresponding to the weekly summary task to be executed is determined based on the start time. Whether the task execution operation corresponding to the weekly summary task to be executed is triggered is determined by determining whether a corresponding record exists in the preset task execution information table. Similarly, the specific process of performing data aggregation at each time granularity can be understood. The preset task execution information table is a data table that stores the task execution records of the pending summary task.

[0073] This embodiment uses a Work Stealing Pool, also known as a task stealing thread pool. This approach allows the main thread to wait for the child thread to complete execution. Different time granularities correspond to different child threads. In each child thread, the source data table is first queried for data to be aggregated, grouped by time range. The retrieved data to be aggregated is then saved to the destination data table. After the task completes, the corresponding execution record is saved to the preset task execution information table in the database.

[0074] It is further understood that the specific execution process of the to-be-executed summary tasks at different time granularities is consistent, but the specific implementation process may differ due to the difference in the corresponding task information. The determining the corresponding to-be-aggregated data from the source data table based on the summary time range and saving the to-be-aggregated data to the destination data table includes: determining the corresponding to-be-aggregated data from the source data table based on the summary time range and the source field pre-recorded in the task information, and saving the to-be-aggregated data to the destination data table.

[0075] Step S14: determining corresponding data to be summarized from the source data table based on the summary time range and saving the data to be summarized to the destination data table to complete the summary operation on the data to be summarized.

[0076] In this embodiment, when determining the corresponding data to be aggregated from the source data table based on the aggregation time range, the specific implementation process includes: using the aggregation time range and the aggregation algorithm pre-recorded in the task information and the source field to obtain the corresponding data to be aggregated; executing the first data judgment operation pre-recorded in the task information to determine whether the primary key of the data to be aggregated is repeated in the target data table, and if so, performing the corresponding data update; executing the second data judgment operation pre-recorded in the task information to determine whether the data to be aggregated is a timestamp, and if so, performing a query for the corresponding time range; executing the third data judgment operation pre-recorded in the task information to determine whether the data to be aggregated is a device ID, and if so, performing the corresponding data grouping operation; using the target field pre-recorded in the task information to save the data to be aggregated to the target data table. In another specific embodiment, the aggregation algorithm is not pre-configured, and the average value is taken in this case. Among them, if the data to be aggregated is a device ID, the data is grouped based on this field as the grouping basis.

[0077] In this embodiment, after completing the aggregation operation on the data to be aggregated, the process further includes: determining a corresponding task execution record based on the table name of the source data table corresponding to the aggregation task to be executed, the table name of the destination data table, the corresponding time granularity, and the aggregation time range, and saving the record to a preset task execution information table. Specifically, the record includes the table name of the source data table, the table name of the destination data table, the corresponding time granularity, and the aggregation start time corresponding to the aggregation time range.

[0078] It can be seen that in this application, first, multiple data sources are configured to obtain several databases, and table names corresponding to the corresponding time granularity are constructed for each data table in the database based on the preset time granularity; then, task information is configured through the task configuration interface to obtain the corresponding summary task to be executed; then, whether the corresponding task execution operation is currently triggered is determined step by step according to the time granularity, and if so, the corresponding summary time range is determined based on the time granularity recorded in the task information, and the source data table and the destination data table pre-recorded in the task information are found from the source database and the destination database; the source database and the destination database are databases determined from the several databases; based on the summary time range, the corresponding data to be summarized are determined from the source data table and the data to be summarized are saved to the destination data table to complete the summary operation of the data to be summarized. This application generates the corresponding summary task to be executed based on the time granularity and summarizes it step by step according to the time granularity, which can effectively reduce the repeated development caused by the addition of new data tables when summarizing data in the existing technology, reduce the amount of summarized data, and improve the summary efficiency and flexibility.

[0079] As can be seen from the previous embodiment, this application determines whether the corresponding task execution operation is currently triggered according to the time granularity. To this end, this embodiment will next describe in detail the process of determining according to the time granularity step by step. Figure 3 As shown, an embodiment of the present invention discloses a data aggregation method with multiple time granularities, including:

[0080] Step S21: Determine whether the current time is 0 minutes and 0 seconds. If so, execute the corresponding summary task to be executed at the hourly granularity.

[0081] Step S22: When the execution of the summary task to be executed at the hourly granularity is completed, determine whether there is a corresponding record in the preset task execution information table based on the first summary start time corresponding to the summary task to be executed at the daily granularity; if not, execute the summary task to be executed at the daily granularity.

[0082] Step S23: When the execution of the summary task to be executed at the daily granularity is completed, determine whether there is a corresponding record in the preset task execution information table based on the second summary start time corresponding to the summary task to be executed at the weekly granularity; if not, execute the summary task to be executed at the weekly granularity.

[0083] Step S24: When the weekly summary task to be executed is completed, determine whether there is a corresponding record in the preset task execution information table based on the third summary start time corresponding to the monthly summary task to be executed. If not, execute the monthly summary task to be executed.

[0084] Step S25: When the monthly granularity summary task to be executed is completed, determine whether there is a corresponding record in the preset task execution information table based on the fourth summary start time corresponding to the quarterly granularity summary task to be executed; if not, execute the quarterly granularity summary task to be executed.

[0085] Step S26: When the execution of the summary task to be executed at the quarterly granularity is completed, determine whether there is a corresponding record in the preset task execution information table based on the fifth summary start time corresponding to the summary task to be executed at the annual granularity; if not, execute the summary task to be executed at the annual granularity.

[0086] Among them, the specific processes of the above steps S21 and S26 can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0087] Thus, in the embodiment of the present application, by starting the preset scheduled task, the corresponding task execution operation is carried out step by step according to the time granularity. The summary tasks to be executed at the hourly granularity, daily granularity, weekly granularity, monthly granularity, quarterly granularity and annual granularity are executed in sequence. In this way, it can be ensured that when the summary tasks to be executed at the current time granularity are executed, the summary tasks to be executed corresponding to the next level of time granularity have all been completed, thereby ensuring data integrity.

[0088] See also Figure 4 As shown, the embodiment of the present application also discloses a data aggregation device with multiple time granularities, including:

[0089] A database configuration module 11 is used to configure multiple data sources to obtain several databases;

[0090] A table name construction module 12 is used to construct a table name corresponding to the corresponding time granularity for each data table in the database based on a preset time granularity;

[0091] The task configuration module 13 is used to configure task information through the task configuration interface to obtain the corresponding summary task to be executed;

[0092] The task execution module 14 is configured to determine, step by step according to the time granularity, whether a corresponding task execution operation is currently triggered. If so, the module determines a corresponding summary time range based on the time granularity recorded in the task information, and searches for a source data table and a destination data table pre-recorded in the task information from a source database and a destination database; the source database and the destination database are databases determined from the plurality of databases.

[0093] The data aggregation module 15 is configured to determine corresponding data to be aggregated from the source data table based on the aggregation time range and save the data to be aggregated to the destination data table, so as to complete the aggregation operation on the data to be aggregated.

[0094] Among them, for more specific working processes of the above modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0095] It can be seen that in this application, first, multiple data sources are configured to obtain several databases, and table names corresponding to the corresponding time granularity are constructed for each data table in the database based on the preset time granularity; then, task information is configured through the task configuration interface to obtain the corresponding summary task to be executed; then, whether the corresponding task execution operation is currently triggered is determined step by step according to the time granularity, and if so, the corresponding summary time range is determined based on the time granularity recorded in the task information, and the source data table and the destination data table pre-recorded in the task information are found from the source database and the destination database; the source database and the destination database are databases determined from the several databases; based on the summary time range, the corresponding data to be summarized are determined from the source data table and the data to be summarized are saved to the destination data table to complete the summary operation of the data to be summarized. This application generates the corresponding summary task to be executed based on the time granularity and summarizes it step by step according to the time granularity, which can effectively reduce the repeated development caused by the addition of new data tables when summarizing data in the existing technology, reduce the amount of summarized data, and improve the summary efficiency and flexibility.

[0096] In some specific embodiments, the data aggregation module 15 may specifically include:

[0097] The data aggregation unit is configured to determine corresponding data to be aggregated from the source data table based on the aggregation time range and the source field pre-recorded in the task information, and save the data to be aggregated to the destination data table.

[0098] In some specific embodiments, the multi-time granularity data aggregation device may further include:

[0099] A data value obtaining unit, configured to obtain a value from the source field using the summary time range and the aggregation algorithm pre-recorded in the task information to obtain the corresponding data to be summarized;

[0100] A first data judgment unit is used to perform a first data judgment operation pre-recorded in the task information to judge whether the primary key of the data to be summarized is repeated in the target data table, and if so, to perform corresponding data update;

[0101] The second data judgment unit executes the second data judgment operation pre-recorded in the task information to judge whether the data to be aggregated is a timestamp, and if so, performs a query within a corresponding time range;

[0102] The third data judgment unit executes the third data judgment operation pre-recorded in the task information to judge whether the data to be aggregated is a device ID, and if so, performs a corresponding data grouping operation;

[0103] The data saving unit is used to save the data to be summarized into the destination data table by using the destination field pre-recorded in the task information.

[0104] In some specific embodiments, the task execution module 14 may specifically include:

[0105] The scheduled task startup unit is used to start the preset scheduled task to regularly determine whether the corresponding task execution operation is triggered at the current time starting from 0:00 of each hour according to the time granularity.

[0106] In some specific embodiments, the multi-time granularity data aggregation device may further include:

[0107] The record saving unit is used to determine the corresponding task execution record based on the table name of the source data table corresponding to the summary task to be executed, the table name of the target data table, the corresponding time granularity and the summary time range, and save the record to the preset task execution information table.

[0108] In some specific embodiments, the task execution module 14 may specifically include:

[0109] The first task execution unit is used to determine whether the current time is zero minutes and zero seconds, and if so, execute the corresponding summary task to be executed at the hourly granularity;

[0110] The second task execution unit is configured to determine, when the execution of the summary task to be executed at the hourly granularity is completed, whether there is a corresponding record in the preset task execution information table based on the first summary start time corresponding to the summary task to be executed at the daily granularity, and if not, execute the summary task to be executed at the daily granularity;

[0111] A third task execution unit is configured to determine, when the execution of the to-be-executed summary task of the daily granularity is completed, whether there is a corresponding record in the preset task execution information table based on the second summary start time corresponding to the to-be-executed summary task of the weekly granularity, and if not, execute the to-be-executed summary task of the weekly granularity;

[0112] A fourth task execution unit is configured to, when the execution of the weekly summary task to be executed is completed, determine whether a corresponding record exists in the preset task execution information table based on the third summary start time corresponding to the monthly summary task to be executed, and if not, execute the monthly summary task to be executed;

[0113] A fifth task execution unit is configured to, when the execution of the to-be-executed summary task of the monthly granularity is completed, determine, based on the fourth summary start time corresponding to the to-be-executed summary task of the quarterly granularity, whether a corresponding record exists in the preset task execution information table, and if not, execute the to-be-executed summary task of the quarterly granularity;

[0114] The sixth task execution unit is used to determine whether there is a corresponding record in the preset task execution information table based on the fifth summary start time corresponding to the summary task to be executed at the annual granularity when the execution of the summary task to be executed at the quarterly granularity is completed, and if not, execute the summary task to be executed at the annual granularity.

[0115] In some specific embodiments, the table name construction module 12 may further include:

[0116] It indicates a suffix adding unit, which is used to construct a corresponding table name suffix based on a preset time granularity, and add the table name suffix to the table name of each data table in the current database to obtain a table name corresponding to the corresponding time granularity; wherein the table name suffix corresponds to the time granularity one by one.

[0117] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0118] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the multi-time granularity data aggregation method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0119] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0120] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0121] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, and can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the multi-time granularity data aggregation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0122] Furthermore, this application discloses a computer-readable storage medium for storing a computer program; when executed by a processor, the computer program implements the aforementioned multi-time granularity data aggregation method. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be further described here.

[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0124] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0126] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0127] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data aggregation method with multiple time granularities, characterized in that: include: Configuring multiple data sources to obtain multiple databases, and constructing table names corresponding to the corresponding time granularities for each data table in the database based on preset time granularities; Configure task information through the task configuration interface to obtain the corresponding summary tasks to be executed; Determine whether a corresponding task execution operation is currently triggered according to the time granularity, and if so, determine a corresponding summary time range based on the time granularity recorded in the task information, and search the source data table and the destination data table pre-recorded in the task information from the source database and the destination database; the source database and the destination database are databases determined from the plurality of databases; Determining corresponding data to be summarized from the source data table based on the summary time range and saving the data to be summarized to the destination data table to complete the summary operation on the data to be summarized; Among them, also include: When filtering data tables based on time granularity, if the selected time granularity is daily, tables with names containing the suffix _hour are filtered out from the source data table, and tables with names containing the suffix _day are filtered out from the destination data table. If the selected time granularity is hourly, tables without time suffixes are filtered out from the source data table, and tables with names containing the suffix _hour are filtered out from the destination data table to determine the aggregation tasks to be executed. The determining corresponding data to be summarized from the source data table based on the summary time range and saving the data to be summarized to the destination data table includes: Determining corresponding data to be summarized from the source data table based on the summary time range and the source field pre-recorded in the task information, and saving the data to be summarized to the destination data table; Also includes: Utilize the aggregation time range and the aggregation algorithm pre-recorded in the task information and the source field to obtain the corresponding data to be aggregated; Execute the first data judgment operation pre-recorded in the task information to determine whether the primary key of the data to be summarized is repeated in the target data table, and if so, perform corresponding data update; Executing a second data determination operation pre-recorded in the task information to determine whether the data to be aggregated is a timestamp, and if so, performing a query within a corresponding time range; executing a third data determination operation pre-recorded in the task information to determine whether the data to be aggregated is a device ID, and if so, performing a corresponding data grouping operation; Using the pre-recorded destination field in the task information, the data to be aggregated is saved in the destination data table; The step of judging whether a corresponding task execution operation is currently triggered according to the time granularity includes: Start the preset scheduled task to regularly determine whether the corresponding task execution operation is triggered at the time granularity starting from 0:00 of each hour; The step of judging whether a corresponding task execution operation is currently triggered according to the time granularity includes: Determine whether the current time is 0 minutes and 0 seconds. If so, execute the corresponding hourly granularity pending summary task; When the execution of the summary task to be executed at the hourly granularity is completed, judging whether there is a corresponding record in the preset task execution information table based on the first summary start time corresponding to the summary task to be executed at the daily granularity, if not, executing the summary task to be executed at the daily granularity; When the execution of the summary task to be executed at the daily granularity is completed, judging whether there is a corresponding record in the preset task execution information table based on the second summary start time corresponding to the summary task to be executed at the weekly granularity, and if not, executing the summary task to be executed at the weekly granularity; When the weekly summary task to be executed is completed, judging whether there is a corresponding record in the preset task execution information table based on the third summary start time corresponding to the monthly summary task to be executed, and if not, executing the monthly summary task to be executed; When the execution of the to-be-executed summary task of the monthly granularity is completed, judging whether there is a corresponding record in the preset task execution information table based on the fourth summary start time corresponding to the to-be-executed summary task of the quarterly granularity, and if not, executing the to-be-executed summary task of the quarterly granularity; When the execution of the summary task to be executed at the quarterly granularity is completed, it is determined whether there is a corresponding record in the preset task execution information table based on the fifth summary start time corresponding to the summary task to be executed at the annual granularity. If not, the summary task to be executed at the annual granularity is executed.

2. The multi-time granularity data aggregation method according to claim 1, characterized in that: After the aggregation operation of the data to be aggregated is completed, the method further includes: Based on the table name of the source data table corresponding to the summary task to be executed, the table name of the destination data table, the corresponding time granularity and the summary time range, a corresponding task execution record is determined and the record is saved in a preset task execution information table.

3. The data aggregation method of multiple time granularities according to claim 1 or 2, characterized in that: The step of constructing a table name corresponding to the corresponding time granularity for each data table in the database based on the preset time granularity includes: A corresponding table name suffix is ​​constructed based on a preset time granularity, and the table name suffix is ​​added to the table name of each data table in the current database to obtain a table name corresponding to the corresponding time granularity; wherein the table name suffix corresponds to the time granularity one by one.

4. A data aggregation device with multiple time granularities, characterized in that: include: Database configuration module, used to configure multiple data sources to obtain several databases; A table name construction module, configured to construct a table name corresponding to a corresponding time granularity for each data table in the database based on a preset time granularity; The task configuration module is used to configure task information through the task configuration interface to obtain the corresponding summary tasks to be executed; a task execution module, configured to determine, step by step according to the time granularity, whether a corresponding task execution operation is currently triggered; if so, to determine a corresponding summary time range based on the time granularity recorded in the task information, and to search a source data table and a destination data table pre-recorded in the task information from a source database and a destination database; the source database and the destination database being databases determined from the plurality of databases; A data aggregation module is used to determine corresponding data to be aggregated from the source data table based on the aggregation time range and save the data to be aggregated to the destination data table to complete the aggregation operation of the data to be aggregated; The multi-time granularity data aggregation device is further configured to: in the process of filtering data tables according to time granularity, when the selected time granularity is day, filter out tables whose table names contain the suffix _hour from the source data table, and filter out tables whose table names contain the suffix _day from the destination data table; when the selected time granularity is hour, filter out tables without time suffixes from the source data table, and filter out tables whose table names contain the suffix _hour from the destination data table, so as to determine the aggregation tasks to be executed; The data aggregation module includes: a data aggregation unit, configured to determine corresponding data to be aggregated from the source data table based on the aggregation time range and the source field pre-recorded in the task information, and save the data to be aggregated to the destination data table; The multi-time granularity data aggregation device further includes: A data value obtaining unit, configured to obtain a value from the source field using the summary time range and the aggregation algorithm pre-recorded in the task information to obtain the corresponding data to be summarized; A first data judgment unit is used to perform a first data judgment operation pre-recorded in the task information to judge whether the primary key of the data to be summarized is repeated in the target data table, and if so, to perform corresponding data update; The second data judgment unit executes the second data judgment operation pre-recorded in the task information to judge whether the data to be aggregated is a timestamp, and if so, performs a query within a corresponding time range; The third data judgment unit executes the third data judgment operation pre-recorded in the task information to judge whether the data to be aggregated is a device ID, and if so, performs a corresponding data grouping operation; A data storage unit, configured to save the data to be aggregated into the destination data table using the destination field pre-recorded in the task information; The task execution module includes: The scheduled task startup unit is used to start the preset scheduled task, and regularly judge whether the corresponding task execution operation is triggered according to the time granularity from 0:00 of each hour; The task execution module includes: The first task execution unit is used to determine whether the current time is zero minutes and zero seconds, and if so, execute the corresponding summary task to be executed at the hourly granularity; The second task execution unit is configured to determine, when the execution of the summary task to be executed at the hourly granularity is completed, whether there is a corresponding record in the preset task execution information table based on the first summary start time corresponding to the summary task to be executed at the daily granularity, and if not, execute the summary task to be executed at the daily granularity; A third task execution unit is configured to determine, when the execution of the to-be-executed summary task of the daily granularity is completed, whether there is a corresponding record in the preset task execution information table based on the second summary start time corresponding to the to-be-executed summary task of the weekly granularity, and if not, execute the to-be-executed summary task of the weekly granularity; A fourth task execution unit is configured to, when the execution of the weekly summary task to be executed is completed, determine whether a corresponding record exists in the preset task execution information table based on the third summary start time corresponding to the monthly summary task to be executed, and if not, execute the monthly summary task to be executed; A fifth task execution unit is configured to, when the execution of the to-be-executed summary task of the monthly granularity is completed, determine, based on the fourth summary start time corresponding to the to-be-executed summary task of the quarterly granularity, whether a corresponding record exists in the preset task execution information table, and if not, execute the to-be-executed summary task of the quarterly granularity; The sixth task execution unit is used to determine whether there is a corresponding record in the preset task execution information table based on the fifth summary start time corresponding to the summary task to be executed at the annual granularity when the execution of the summary task to be executed at the quarterly granularity is completed, and if not, execute the summary task to be executed at the annual granularity.

5. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the multi-time granularity data aggregation method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the multi-time granularity data aggregation method according to any one of claims 1 to 3.

Images