A distributed summary table data generation and query method

Through the distributed system, the problem of long return time of database aggregation query is solved, and the result is quickly generated and returned aggregated query results are realized, which improves query efficiency.

CN117827900BActive Publication Date: 2024-11-05TECH & ENG CENT FOR SPACE UTILIZATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311694635.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-11
Publication Date
2024-11-05
Estimated Expiration
2043-12-11

AI Technical Summary

Technical Problem

The prior art cannot quickly generate summary data during database aggregation query, resulting in the return time of the aggregate query result being too long to meet the needs of efficient query.

Method used

Using the summary table data generation and query method for distributed systems, the node server receives the original timing data and creates processing tasks through the task generation node server. Multiple tasks consume the node server generates the summary table data and saves it to the distributed timing database. The query server is based on the query. The parameters are converted into conditions and returned the result, and the distributed lock is used to prevent duplicate processing. The time span is divided by hierarchical method to quickly generate summary data.

Benefits of technology

It realizes the rapid generation and return of aggregate query results during database aggregation query, improves the efficiency of aggregate query, and solves the problem of long-term generation of abstract data in the prior art.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117827900B_ABST
    Figure CN117827900B_ABST
Patent Text Reader

Abstract

The present invention provides a distributed summary table data generation and query method, the method comprising: first, a task generation node server receives original time series data. Then, the task generation node server creates a processing task according to the device name in the original time series data and the acquisition time of the original time series data. Multiple task consumption node servers can generate summary table data according to the processing tasks, and save the summary table data to a distributed time series database. Furthermore, the query server can convert the received query parameters into query conditions, and determine and return the query results in the distributed time series database according to the query conditions. The present invention is based on a distributed system, and generates summary table data according to the processing tasks through multiple task consumption node servers. In this way, summary data can be generated quickly, and when an aggregate query is performed on the database, the results of the aggregate query can be quickly returned, thereby improving the efficiency of the aggregate query.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of aggregate query of database data, and in particular to a distributed summary table data generation and query method. Background Art

[0002] After data is stored in the database, it is usually aggregated and queried based on business needs. For example, the banking system will calculate the total deposit amount of all current users and the maximum deposit amount of a single user based on the user deposit table in the bank database. Such aggregate query operations usually scan the entire table data and perform the required aggregate calculations without any additional technical processing. This not only requires the database to have powerful computing performance, but also consumes a lot of time for aggregate calculations.

[0003] Usually, methods such as index optimization, partitioning and table division, pre-aggregation and summary tables, materialized views, column storage databases, memory optimization, and pre-compiled long structured query language (SQL) can be used to process aggregate queries. Specifically, for the pre-aggregation and summary table methods, the summary table data can be generated by using the summary forest method based on the segment tree algorithm, or by using M4 data preprocessing and a summary tree based on a B+ tree to generate summary table data.

[0004] However, the above tree structure is used to organize summary data, which will have a large time complexity when searching for data and will consume a lot of time. And due to the problem of data expansion rate, it cannot be guaranteed that all required data has been generated. It can be seen that the existing related technologies cannot quickly generate summary data, which leads to the problem that when performing aggregate queries on the database, the results of the aggregate queries cannot be quickly returned. Summary of the invention

[0005] The technical problem to be solved by the present invention is that when performing an aggregate query on a database, the result of the aggregate query cannot be quickly returned.

[0006] In order to solve the above technical problems, the present invention provides a distributed summary table data generation and query method. The specific technical solution is as follows:

[0007] The present invention provides a distributed summary table data generation and query method, which can be applied to a distributed aggregation query system, which includes: a task generation node server, a plurality of task consumption node servers and a query server. Specifically, the method includes: first, the task generation node server receives the original time series data, and the original time series data includes: a device name. Then, the task generation node server creates a processing task according to the device name and the acquisition time of the original time series data, and the processing task is used to instruct the task consumption node server to generate summary table data. Next, the plurality of task consumption node servers can generate summary table data according to the processing tasks, and save the summary table data to a distributed time series database. Furthermore, the query server can convert the received query parameters into query conditions, and determine and return the query results in the distributed time series database according to the query conditions.

[0008] In the method, the task generation node server can create a processing task according to the device name and the acquisition time of the original time series data. Multiple task consumption node servers generate summary table data according to the processing tasks, and save the summary table data to the distributed time series database. The query server can convert the received query parameters into query conditions, and determine and return the query results in the distributed time series database according to the query conditions. The solution of the present invention is based on a distributed system, and multiple task consumption node servers generate summary table data according to the processing tasks. In this way, summary data can be generated quickly, and when an aggregate query is performed on the database, the results of the aggregate query can be quickly returned, thereby improving the efficiency of the aggregate query.

[0009] In an optional implementation, the task generation node server creates a processing task according to the device name and the collection time of the original time series data, specifically including: first, the task generation node server monitors the received original time series data and extracts the device name. Then, the task generation node server determines the task generation time corresponding to the device according to the device name, and the task generation time is determined according to the data collection frequency of the device and the first preset data volume, and the device corresponds to the device name. Finally, the task generation node server creates a processing task corresponding to the device when the collection time of the original time series data meets the task generation time.

[0010] In an optional implementation, the above-mentioned multiple task consumption node servers generate summary table data according to the processing tasks, and save the summary table data to the distributed time series database, which specifically includes: first, the task consumption node server queries the task information of the processing task, and the task information of the processing task includes: task start time, task end time and device name. Then, the task consumption node server applies for a first distributed lock according to the task information of the processing task, and the first distributed lock is used to prevent the processing task from being repeatedly processed in the same time period. Finally, when the task consumption node server applies for the first distributed lock, it generates corresponding summary table data according to the processing task.

[0011] In an optional implementation, when the task consumption node server applies for the first distributed lock, after generating the corresponding summary table data according to the processing task, the method further includes: the task consumption node server applies for the second distributed lock. When the task consumption node server applies for the second distributed lock and the second distributed lock is the same as the first distributed lock, the summary table data is saved in the distributed time series database. In this way, it is possible to prevent multiple task consumption node servers from processing the same processing task, that is, to prevent the same processing task from being processed repeatedly.

[0012] In an optional implementation, the task consumption node server generates corresponding summary table data according to the processing task when applying for the first distributed lock, specifically including: first, the task consumption node server obtains the historical calculation result corresponding to the device according to the device name. Then, the task consumption node server determines the original time series data corresponding to the device according to the task start time and task end time. Finally, the task consumption node server generates summary table data according to the original time series data corresponding to the device and the historical calculation results.

[0013] In an optional implementation, the summary table data includes: L levels of feature point data, the L levels of feature point data including 2 (L-1)* at least one aggregation information of n original time series data, wherein L is a positive integer greater than or equal to 1, and n is a second preset data volume. The original time series data also includes: at least one parameter and the value corresponding to the parameter. Then the task consumption node server generates summary table data according to the original time series data corresponding to the device and the historical calculation results, specifically including: first, the task consumption node server obtains the L-level data structure corresponding to the device, the L-level data structure corresponds to the L-level feature point data one by one, and the data structure includes: calculation start time, calculation end time and aggregation function. Then, the task consumption node server sorts the L-level data structure in the order of the calculation end time from small to large, and, when the calculation end time is the same, sorts the L-level data structure in the order of the calculation start time from small to large, and obtains the sorted L-level data structure. Next, the task consumption node server calculates the aggregation function in each level of the data structure in the L-level data structure according to the value corresponding to the parameter in the original time series data and the historical calculation results, and obtains the corresponding L-level feature point data. Finally, when the calculation end time of the data structure is greater than the collection time of the original time series data, the task consumption node server saves the feature point data of the level corresponding to the calculation end time. In this implementation, the task consumption node server can divide the size of the time span based on a hierarchical manner, determine the feature point data of different levels, and thus obtain the summary table data. In this way, the aggregation information of the original time series data can be accurately determined through the feature point data of different levels.

[0014] In an optional implementation, the above-mentioned multiple task consumption node servers generate summary table data according to the processing tasks, and save the summary table data to the distributed time series database, and also include: the task consumption node server saves the feature point data according to the device name, the level of the feature point data and the time partition key as the primary key. Among them, the expression of the time partition key is: pk = dataTime / (MaxCapacity / EachItemSpace*n*Freq*2 (R-1) ); where pk is the time partition key, dataTime is the time of the feature point data, MaxCapacity is the preset maximum capacity to ensure query performance, EachItemSpace is the preset space occupied by the device when calculating all aggregation functions, n is the second preset data volume, Freq is the data collection frequency of the device, and R is the level of the feature point data. In this way, it not only meets the demand for device space occupation, but also facilitates the query server to quickly locate the location of the feature point data in the distributed time series database according to the primary key, thereby improving the efficiency of aggregate query.

[0015] In an alternative implementation, the above-mentioned distributed aggregation query system further includes: a distributed configuration center server. Then the method further includes: the distributed configuration center server sends a configuration file to the task-consuming node server, and the configuration file is used to indicate the aggregation function to be calculated. The task-consuming node server receives the configuration file and calculates the aggregation function according to the configuration file to obtain summary table data. In this implementation, it can be ensured that after the configuration is updated in the distributed configuration center server, the task-consuming node server can immediately receive the relevant configuration file, so as to ensure that the task-consuming node server can accurately determine the corresponding aggregation information according to the aggregation function and generate summary table data.

[0016] In an alternative implementation, the above-mentioned query parameters include: the total query start time, the total query end time, and the query device name. Then the above-mentioned query server converts the received query parameters into query conditions, and determines and returns the query result in the distributed time-series database according to the query conditions, specifically including: the query server determines the query time span according to the total query start time and the total query end time, and the query time span is the difference between the total query end time and the total query start time. The query server determines the basic time window according to the data collection frequency and the second preset data volume of the device corresponding to the query device name, and the basic time window is the product of the data collection frequency of the device corresponding to the query device name and the second preset data volume. The query server determines the query level of the feature point data according to the query time span and the basic time window, and the query level of the feature point data satisfies the following expression: when X*n*window < timespan ≤ (X + 1)*n*window, the query level of the feature point data is X + 1, where, timespan is the query time span, window is the basic time window, and n is the second preset data volume. The query server determines the time partition key query range according to the total query start time, the total query end time, the basic time window, and the query level of the feature point data, and the time partition key query range satisfies the following expression: pk start = QuerystartTime / (MaxCapacity / EachItemSpace*window*2 (l-1) ) ~ pk end = QueryendTime / (MaxCapacity / EachItemSpace*window*2 (l-1) ). Where, pk start is the start range of the time partition key query, pk endis the end range of the time partition key query, l is the query level of the feature point data, QuerystartTime is the total query start time, and QueryendTime is the total query end time. The query server determines and returns the query result in the distributed time series database based on the query device name, the query level of the feature point data, the time partition key range, the total query start time, and the total query end time.

[0017] In an optional implementation, the query server determines and returns the query result in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time and the total query end time, specifically including: the query server obtains the first feature point data result by querying in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time and the total query end time, and the first feature point data result is arranged in order from small to large according to the start time. The query server determines whether the smallest start time in the first feature point data result is equal to the total query start time, and whether the largest end time in the first feature point data result is equal to the total query end time. When the smallest start time in the first feature point data result is equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, the query server uses the first feature point data result as the query result. When the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and the query level of the feature point data is greater than 1, the query server uses the total query start time as the first sub-query start time, the smallest start time as the first sub-query end time, and the query level of the feature point data minus 1 as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the second feature point data result. Use the largest end time as the second sub-query start time, the total query end time as the second sub-query end time, and the query level of the feature point data minus 1 as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the third feature point data result. Until the smallest start time in the second feature point data result is equal to the total query start time, and the largest end time in the third feature point data result is equal to the total query end time, the first feature point data result, the second feature point data result, and the third feature point data result are used as query results. The query server finds that the smallest start time in the first feature point data result is equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and when the query level of the feature point data is greater than 1, the largest end time is used as the third sub-query start time, the total query end time is used as the third sub-query end time, and the query level of the feature point data minus 1 is used as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the fourth feature point data result, until the largest end time in the fourth feature point data result is equal to the total query end time, and the first feature point data result and the fourth feature point data result are used as the query results.When the query server finds that the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, and when the query level of the feature point data is greater than 1, the total query start time is used as the fourth sub-query start time, the smallest start time is used as the fourth sub-query end time, and the query level of the feature point data minus 1 is used as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the fifth feature point data result, until the smallest start time in the fifth feature point data result is equal to the total query start time, and the first feature point data result and the fifth feature point data result are used as the query results. When the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and the query level of the feature point data is equal to 1, the query server uses the total query start time as the fifth sub-query start time, and the smallest start time as the fifth sub-query end time, queries the original time series data to obtain the first query time series data, and calculates the sixth feature point data result according to the first query time series data and the aggregation function; uses the largest end time as the sixth sub-query start time, and the total query end time as the sixth sub-query end time, queries the original time series data to obtain the second query time series data, and calculates the seventh feature point data result according to the second query time series data and the aggregation function. . The first feature point data result, the sixth feature point data result, and the seventh feature point data result are used as query results. When the smallest start time in the first feature point data result is equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and the query level of the feature point data is equal to 1, the query server uses the largest end time as the seventh query start time, and the total query end time as the seventh query end time, queries the original time series data to obtain the third query time series data, and calculates the eighth feature point data result according to the third query time series data and the aggregation function. The first feature point data result and the eighth feature point data result are used as the query result. When the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, and the query level of the feature point data is equal to 1, the query server uses the total query start time as the eighth query start time, and the smallest start time as the eighth query end time, queries the original time series data to obtain the fourth query time series data, and calculates the ninth feature point data result according to the fourth query time series data and the aggregation function. The first feature point data result and the ninth feature point data result are used as the query result. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A schematic diagram of the structure of a distributed aggregation query system provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a method flow of a distributed summary table data generation and query method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the structure of a task generation node server provided in an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of a process for creating a task processing task by a task generation node server provided in an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of a task consumption node server provided by an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of a flow chart of a method for generating summary data by a task consumption node server provided in an embodiment of the present invention;

[0024] Figure 7 A schematic diagram of a flow principle for generating summary data by a task consumption node server provided in an embodiment of the present invention;

[0025] Figure 8 A schematic diagram of the structure of a query server provided by an embodiment of the present invention;

[0026] Fig. 9 A schematic diagram of a flow chart of a method for determining a query result by a query server provided in an embodiment of the present invention;

[0027] Fig.10 A schematic diagram of the time span of the first feature point data result provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0029] After data is stored in the database, it is usually aggregated and queried based on business needs. For example, the banking system will calculate the total deposit amount of all current users and the maximum deposit amount of a single user based on the user deposit table in the bank database. Such aggregate query operations usually scan the entire table data and perform the required aggregate calculations without any additional technical processing. This not only requires the database to have powerful computing performance, but also consumes a lot of time for aggregate calculations.

[0030] Aggregate queries can usually be processed using methods such as index optimization, partitioning and table division, pre-aggregation and summary tables, materialized views, column storage databases, memory optimization, and pre-compiled long structured query language (SQL).

[0031] Specifically, index optimization can design appropriate indexes for tables and fields involved in aggregate calculations, so as to reduce the loss during data aggregate query. Partitioning and table division can decompose a large business data table into several tables with the same structure, and determine the location where each data needs to be inserted through certain algorithm logic. In this way, the amount of data in each table can be controlled within a certain range. Therefore, when performing aggregate calculations, the amount of data for a single query can be reduced. Pre-aggregation and summary tables can calculate the aggregate information that may be needed in the future in advance when data is inserted according to business needs, and store it in a separate table. In this way, when there is a need for aggregate query, only retrieval and simple calculations are required to obtain the required results. Materialized views can maintain the required business data directly into a view according to logical relationships when data is inserted. When performing aggregate queries, the required data can be directly retrieved from the view for calculation. Column storage databases allow only the required columns to be read without retrieving the entire row of data. Column storage databases are usually stored adjacently on disk, and can also efficiently utilize the disk read speed when retrieving data from the disk. Memory optimization can load the data required for calculation into memory, and increase the calculation speed by calculating in memory. Precompiling long SQL statements allows the database to parse and optimize a SQL statement in advance and cache the SQL statement. Therefore, when similar queries need to be executed multiple times, precompiling SQL statements can significantly improve query performance. However, the cost of aggregate calculations still requires real-time calculation of the queried data.

[0032] Among them, for the pre-aggregation and summary table method, specifically, the summary table data can be generated by using the summary forest method based on the segment tree algorithm, or by using M4 data preprocessing and a summary tree based on a B+ tree to realize the generation of summary table data.

[0033] However, the above tree structure is used to organize summary data, which will have a large time complexity when searching for data and will consume a lot of time. And due to the problem of data expansion rate, it cannot be guaranteed that all required data has been generated. It can be seen that the existing related technologies cannot quickly generate summary data, which leads to the problem that when performing aggregate queries on the database, the results of the aggregate queries cannot be quickly returned.

[0034] In order to solve the above problems, an embodiment of the present invention provides a distributed summary table data generation and query method, in which a task generation node server can create a processing task according to the device name and the acquisition time of the original time series data. Multiple task consumption node servers generate summary table data according to the processing tasks, and save the summary table data to a distributed time series database. The summary table data includes feature point data of multiple levels. The query server can convert the received query parameters into query conditions, and determine and return the query results in the distributed time series database according to the query conditions. In this way, the scheme of the present invention is based on a distributed system, and multiple task consumption node servers generate summary table data according to the processing tasks, and divide the size of the time span based on a hierarchical manner, determine the feature point data of different levels, and thus obtain summary table data. In this way, summary data can be generated quickly, and when an aggregate query is performed on the database, the results of the aggregate query can be quickly returned, thereby improving the efficiency of the aggregate query.

[0035] The solution provided by the embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0036] The embodiment of the present application provides a distributed summary table data generation and query method, which can be applied to a distributed aggregation query system. Figure 1 A schematic diagram of the structure of a distributed aggregation query system provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the distributed aggregation query system includes: a task generation node server, multiple task consumption node servers and a query server. Among them, the task generation node server is used to receive the original time series data and create processing tasks. The task consumption node server is used to process tasks, generate summary table data and save it in a distributed time series database. The query server is used to determine and return query results in the distributed time series database according to the query conditions.

[0037] For details, see Figure 2 , which is a schematic diagram of a method flow of a distributed summary table data generation and query method provided by an embodiment of the present invention, such as Figure 2 As shown, the distributed summary table data generation and query method provided by the present invention includes the following steps S101-S104:

[0038] S101. The task generation node server receives original time series data.

[0039] Specifically, the task generation node server can receive the pushed original time series data through the Kafka message queue. The original time series data is data arranged in chronological order. In the embodiment of the present application, the original time series data at least includes: device name, at least one parameter and the value corresponding to the parameter.

[0040] Exemplarily, the data structure of the original time series data may be:

[0041] {

[0042] "Device Name 1": {

[0043] "Parameter name 1": "Parameter 1 corresponding value",

[0044] "Parameter name 2": "Parameter 2 corresponding value",

[0045] …

[0046] "Parameter name x": "parameter x corresponding value"

[0047] }

[0048] }

[0049] The original time series data may include data of multiple devices, and each device may include multiple parameters and a value corresponding to each parameter.

[0050] S102: The task generation node server creates a processing task according to the device name and the collection time of the original time series data.

[0051] In the embodiment of the present application, the task generation node server can determine the corresponding device according to the device name in the original time series data, and when the acquisition time of the original time series data reaches the task generation time corresponding to the device, generate the processing task of the device and save it in the task database. The processing task can be used to instruct the task consumption node server to generate the corresponding summary table data.

[0052] S103. Multiple task consumption node servers generate summary table data according to the processed tasks, and save the summary table data into a distributed time series database.

[0053] Specifically, multiple task consumption node servers can obtain different processing tasks respectively, for example, they can obtain processing tasks from the above-mentioned task database. Then, the task consumption node server processes according to the processing tasks, generates corresponding summary table data, and then saves the summary table data to the distributed time series database, so that when performing aggregate queries, it can quickly find and return the results of the aggregate query.

[0054] In some embodiments, in order to prevent multiple task consumption node servers from processing the same processing task, that is, to prevent the same processing task from being processed repeatedly, the task consumption node server can obtain a distributed lock before and after processing each processing task. Specifically, the method provided in the embodiment of the present application also includes: first, the task consumption node server queries the task information of the processing task, and the task information of the processing task includes: task start time, task end time and device name. Then, the task consumption node server applies for a first distributed lock based on the task information of the processing task, and the first distributed lock is used to prevent the processing task from being processed repeatedly in the same period. Next, when the task consumption node server applies for the first distributed lock, it generates corresponding summary table data according to the processing task.

[0055] Furthermore, after the task consumption node server generates the corresponding summary table data according to the processing task, the task consumption node server applies for the second distributed lock again. When the task consumption node server applies for the second distributed lock, and the second distributed lock is the same as the first distributed lock (for example, the same version), the summary table data is saved in the distributed time series database. In this way, the same processing task can be prevented from being repeatedly processed by multiple task consumption node servers, so as to improve the efficiency of the task consumption node server in generating summary table data.

[0056] S104: The query server converts the received query parameters into query conditions, and determines and returns query results in the distributed time series database according to the query conditions.

[0057] In an embodiment of the present application, the query server can convert the query parameters input by the user into query conditions so as to quickly and accurately determine the query results in the distributed time series database. The query parameters include: the total query start time, the total query end time and the query device name. The query conditions include: the query device name, the query level of the feature point data, the time partition key range, the total query start time and the total query end time.

[0058] The distributed summary table data generation and query method provided in the above embodiment of the present application is adopted, based on a distributed system, and summary table data is generated according to processing tasks through multiple task consumption node servers. In this way, summary data can be generated quickly, and when performing aggregate query on the database, the result of the aggregate query can be quickly returned, thereby improving the efficiency of the aggregate query.

[0059] In some embodiments, Figure 3 A schematic diagram of the structure of a task generation node server provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, the task generation node server may include: a loading information module, a monitoring module and a task generation module.

[0060] Figure 4 A flowchart of a method for creating a processing task by a task generation node server provided in an embodiment of the present invention is shown in FIG. Figure 4 As shown, the above S102 may specifically include the following steps S1021-S1024:

[0061] S1021. The task generation node server determines a task generation time corresponding to the device according to a data collection frequency of the device and a first preset data volume.

[0062] In one implementation, this step can be performed by a loading information module in the task generation node server. Specifically, the loading information module can read the device information of all devices in the database and extract the data collection frequency of the device from the device information. Then, the loading information module determines the task generation time corresponding to the device according to the data collection frequency of the device and the first preset data volume. The expression of the task generation time is:

[0063] Timespan = freq*N;

[0064] Among them, Timespan is the task generation time, freq is the data collection frequency of the device, and N is the first preset data volume. Specifically, if the first preset data volume is set too small, the query method will be frequently executed in subsequent query tasks, increasing the pressure on the task generation node server and the distributed time series database. If the first preset data volume is set too large, the processing time of the task consumption node server will increase. Therefore, the first preset data volume can be set according to actual application requirements and prior knowledge. Exemplarily, the first preset data volume can be set to 2000. In this way, when the subsequent task consumption node server queries about 2000 data at one time for processing, the frequency of querying the database and the time to process the query task are relatively reasonable.

[0065] Furthermore, the loading information module can store the determined task generation time corresponding to each device in the memory of the task generation node server in a HashMap data structure. Exemplarily, the task generation time data structure corresponding to the device can be:

[0066]

[0067]

[0068] Timespan is the task generation time in seconds. For example, if the time span of device 1 is 2000, it means that the frequency of device 1 is 1 second / packet. startTime is the starting timestamp of the current task, and lastTime is the time of the last monitored data after the monitoring module is started. When the task generation node server is started, startTime and lastTime are the system time of the current task generation node server.

[0069] S1022. The task generation node server monitors the received original time series data and extracts the device name.

[0070] In one implementation, this step can be performed by a monitoring module in the task generation node server. Specifically, after the loading information module determines to obtain the task generation time corresponding to all devices, the monitoring module can be enabled, and the monitoring module can open the data topic with the Kafka message queue to monitor the received original time series data. After receiving the original time series data, the monitoring module can extract the device name to determine the corresponding task generation time according to the device name.

[0071] S1023. The task generation node server determines the task generation time corresponding to the device according to the device name.

[0072] In one implementation, this step may be performed by a monitoring module in the task generation node server. Further, since the task generation node server stores the task generation time corresponding to each device determined by the loading information module in S1021 in the memory of the task generation node server, the monitoring module may search the memory of the task generation node server according to the device name to determine the device corresponding to the device name and the task generation time corresponding to the device.

[0073] S1024. The task generation node server creates a processing task corresponding to the device when the collection time of the original time series data meets the task generation time.

[0074] In one implementation, this step may be performed by a task generation module in the task generation node server. Specifically, the task generation module may create a processing task corresponding to the device based on the acquisition time of the received original time series data and in the case where the task generation time is met, and save the task in a database (e.g. Figure 1 The task database in the distributed aggregation query system shown).

[0075] Exemplarily, the task generation module can obtain the time difference result by subtracting the startTime in the corresponding task generation time data structure in the memory from the collection time of the original time series data (i.e., the timestamp in the original data). When the time difference result is greater than or equal to the Timespan in the corresponding task generation time data structure in the memory, a corresponding processing task is created and saved in the database. In addition, the task generation module creates a new task generation time data structure using the timestamp of the original time series data as the new startTime. When the above time difference result is less than the Timespan in the corresponding task generation time data structure in the memory, the lastTime in the corresponding task generation time data structure in the memory is updated with the timestamp of the original time series data.

[0076] In some embodiments, Figure 5 A schematic diagram of the structure of a task consumption node server provided by an embodiment of the present invention, such as Figure 5 As shown, the above-mentioned task consumption node server may include: a task query module, a distributed lock application module, a retrieval module and a calculation module.

[0077] Figure 6 A flow chart of a method for generating summary data by a task consumption node server provided in an embodiment of the present invention is as follows: Figure 6 As shown, combined with the process of the task consumption node server acquiring the distributed lock in the above S103, S103 may specifically include the following steps S1031-S1035:

[0078] S1031. The task consumption node server queries the task information of the processing task.

[0079] In one implementation, this step may be performed by a task query module in a task consumption node server. Figure 7 A schematic diagram of the flow principle of generating summary data by a task consumption node server provided in an embodiment of the present invention, such as Figure 7 As shown, specifically, the task query module can query the task information of the processing task through the task database, and the task information of the processing task includes: task start time, task end time and device name.

[0080] S1032: The task consumption node server applies for a first distributed lock according to the task information of the processing task.

[0081] In one implementation, this step can be performed by a distributed lock application module in the task consumption node server. Figure 7As shown, specifically, the distributed lock application module can apply for the first distributed lock to the distributed lock database (such as a Redis memory database) according to the task information of the processing task (such as processing task A). If the first distributed lock application fails, it means that processing task A is being processed by other task consumption node servers, so processing task A is no longer processed. Select the next processing task (such as processing task B), and apply for the first distributed lock again according to the task information of processing task B. If the first distributed lock application is successful, continue to process processing task B. Otherwise, select the next processing task (such as processing task C) to apply for the first distributed lock again until the selected processing task successfully applies for the first distributed lock.

[0082] S1033. When the task consumption node server applies for the first distributed lock, it obtains the historical calculation result corresponding to the device according to the device name.

[0083] In one implementation, this step may be performed by a retrieval module in the task consumption node server. Figure 7 As shown, specifically, the retrieval module can obtain the historical calculation results corresponding to the device (ie, the most recent calculation results of the device) from the distributed cache, so as to calculate the summary table data according to the historical calculation results corresponding to the device and in combination with the original time series data.

[0084] S1034. The task consumption node server determines the original time series data corresponding to the device according to the task start time and task end time.

[0085] In one implementation, this step may be performed by a retrieval module in the task consumption node server. Figure 7 As shown, specifically, the retrieval module can retrieve the original timing data corresponding to the device within the task start time and task end time from the received original timing data based on the time information of the processing task, i.e., the task start time and the task end time, so as to generate the summary table data corresponding to the task processed by the device.

[0086] S1035. The task consumption node server generates summary table data according to the original time series data and historical calculation results corresponding to the device.

[0087] In some embodiments, the summary table data includes: L levels of feature point data, the L levels of feature point data including 2 (L-1) *At least one aggregation information of n original time series data, where L is a positive integer greater than or equal to 1, and n is a second preset data amount.

[0088] Specifically, in the application scenario of the embodiment of the present application, due to the influence of device performance (specifically, for example, browser performance), loading too much data at one time and drawing it into a curve will cause the browser to freeze, and the user cannot obtain valid information of tens of thousands to tens of millions of points from the area of ​​the screen size with the naked eye. Therefore, an upper limit of the curve point display is set according to the application requirements, and the display upper limit will neither affect the performance of the browser nor prevent the user from quickly obtaining valid information from the massive data.

[0089] Usually, when the amount of data in the query result exceeds the upper limit of the amount of data to be displayed, the method of extracting data with equal steps can be used to determine the data to be displayed. For example, if the upper limit of the amount of data displayed by the device is 300 pieces of data and the query result includes 600 pieces of data, then every other piece of data can be extracted and displayed. However, in the case of a query with a larger span, the frequency of extracting data will be too high, making it easy for key information of a certain period of time to be extracted, thereby reducing the accuracy of the query result.

[0090] Therefore, in an embodiment of the present application, when sampling logic appears in the data query (that is, when the data volume of the query result exceeds the upper limit of the display data volume), an aggregate description of the data can be made through feature point data of different levels, and the feature point data includes at least one aggregate information of the original time series data within a period of time.

[0091] Specifically, if the device displays more than the second preset data volume n, the sampling logic will be triggered. The second preset data volume can be set according to actual application requirements and prior knowledge. For example, the second preset data volume can be set to 300. In the embodiment of the present application, the original time series data is described in an aggregated manner through L levels of feature point data. The L level of feature point data includes 2 (L-1) At least one aggregation information of *n original time series data. For example, the feature point data of level 1 is obtained by aggregating n original time series data, the feature point data of level 2 is obtained by aggregating 2*n original time series data, and the feature point data of level 3 is obtained by aggregating 2*n original time series data. 2 *n original time series data are aggregated. In this way, when the amount of feature point data at level 1 is greater than n, n / 2 feature point data at level 1 can be used, and when the amount of feature point data at level 2 is greater than n, n / 2 feature point data at level 3 can be used. In this way, the aggregation information of the original time series data can be accurately determined through feature point data at different levels.

[0092] In some embodiments, the maximum level L of the feature point data can be determined based on the storage volume of the historical data of the device. For example, when the historical data stores 180 days of data, the product of the time span of the feature point data corresponding to the maximum level L and the second preset data volume n is greater than 180 days. The present application does not specifically limit the maximum level L of the feature point data.

[0093] In some embodiments, the data structure of the feature point data includes: device name, time interval, level of feature point data, version number, calculation start time, calculation end time, aggregation function and aggregation information, etc.

[0094] Exemplarily, the data structure of the feature point data may be the following structure:

[0095]

[0096]

[0097] Among them, deviceName is the device name, level is the level of feature point data, interval is the time interval (unit: seconds), StartTime is the calculation start time, EndTime is the calculation end time, specifically, EndTime = StartTime + interval, that is, when the calculated data time is greater than the calculation end time, this data structure should be saved in the distributed time series database. version is the version number. para1 is the aggregation information with parameter name para1, min is the aggregation function of the minimum value, time in min is the occurrence time of the minimum value, and value in min is the specific value of the minimum value. max is the aggregation function of the maximum value, time in max is the occurrence time of the maximum value, and value in max is the specific value of the maximum value. avg is the aggregation function of the average value, value in avg is the specific value of the average value, and count in avg is the amount of data covered by the average value.

[0098] In the embodiment of the present application, the aggregate information used to calculate the aggregate function includes but is not limited to: maximum value, minimum value, mean, median, mode, number of inflection points, variance range, dispersion coefficient, skewness coefficient, kurtosis coefficient and other aggregate information. The aggregate function can be set according to actual application requirements, and this application does not make specific restrictions on this.

[0099] In some embodiments, the data structure of the feature point data of different levels is constructed when the system is initialized and stored in a distributed cache database (e.g., a Redis memory database). After the task consumption node server obtains the processing task of the device and successfully applies for the first distributed lock, the data structure of the feature point data of all levels corresponding to the device will be updated from the distributed cache database cache to the local of the task consumption node server.

[0100] In some embodiments, the above S1035, the task consumption node server generates summary table data according to the original time series data and historical calculation results corresponding to the device, which may specifically include the following steps S10351-S10354:

[0101] S10351. The task consumption node server obtains L levels of data structures corresponding to the device, and the L levels of data structures correspond one-to-one to the L levels of feature point data.

[0102] In one implementation, this step can be performed by a computing module in the task consumption node server. Specifically, the computing module can obtain all L levels of data structures corresponding to the device from a distributed cache database (such as a Redis memory database). The data structure includes at least: calculation start time, calculation end time and aggregation function.

[0103] S10352. The task consumption node server sorts the data structures of L levels in the order of the calculation end time from small to large, and, when the calculation end time is the same, sorts the data structures of L levels in the order of the calculation start time from small to large, to obtain the sorted data structures of L levels.

[0104] In one implementation, this step can be performed by a computing module in a task consumption node server. Specifically, after obtaining all L levels of data structures corresponding to the device, the computing module sorts the L levels of data structures. The sorting rule is: the L levels of data structures are sorted in order of the calculation end time from small to large, and, when the calculation end time is the same, they are sorted in order of the calculation start time from small to large. In this way, it can be ensured that the calculation end time of the data result of the first corresponding level is the earliest.

[0105] In one implementation, an ordered set TreeSet can be used to sort L levels of data structures. TreeSet is a tree-shaped data structure of an ordered set, and the user-preset comparison logic can be used to find the corresponding positions of inserted data structures of different levels, thereby achieving sorting. Exemplarily, there are two levels of data structures in the TreeSet ordered set, and the calculation end time of the first level data structure is 00:42:10 on November 6, 2023, and the calculation end time of the second level data structure is 00:48:29 on November 6, 2023. When inserting the third level data structure, since the calculation end time of the third level data structure is 00:44:53 on November 6, 2023, the third level data structure will be inserted into the TreeSet ordered set. After that, it will be ranked second, thereby realizing that the L levels of data structures are sorted in order from small to large according to the calculation end time.

[0106] S10353. The task consumption node server calculates the aggregation function in each level of the data structure in the L levels according to the values ​​corresponding to the parameters in the original time series data and the historical calculation results, and obtains the corresponding L levels of feature point data in the order of the sorted L levels of data structures.

[0107] In one implementation, this step may be performed by a computing module in the task consumption node server. Specifically, the computing module may traverse each original time series data, and calculate the aggregation function in each level of the data structure according to the values ​​corresponding to the parameters in the original time series data and the historical calculation results.

[0108] Exemplarily, taking the aggregation function as calculating the maximum value as an example, the calculation module can determine whether the value corresponding to the parameter in the original time series data (for example, A) is greater than the historical maximum value calculation result (i.e., the historical calculation maximum value B in the data structure). In the case where the value A corresponding to the parameter in the original time series data is greater than the historical calculation maximum value B in the data structure, the historical calculation maximum value B in the data structure is updated to the value A corresponding to the parameter in the original time series data. In the case where the value A corresponding to the parameter in the original time series data is less than or equal to the historical calculation maximum value B in the data structure, the historical calculation maximum value B in the data structure is kept unchanged.

[0109] S10354. When the calculation end time of the data structure is greater than the collection time of the original time series data currently being processed, the task consumption node server saves the feature point data of the level corresponding to the calculation end time.

[0110] In one implementation, this step can be performed by a computing module in the task consumption node server. Specifically, after traversing each piece of original time series data, the computing module can check whether the calculation end of the data structure of the feature points of each level in the TreeSet ordered set is greater than the acquisition time of the current piece of original time series data. If greater, the feature point data of the corresponding level of the data structure is saved.

[0111] In some embodiments, before S10354, the task consumption node server can also create an empty sequence list in the local memory to facilitate temporary storage of feature point data that needs to be inserted into the distributed time series database after subsequent operations. After the task consumption node server completes the calculation of all the original time series data in S10354, the sequence list is inserted into the distributed time series database. In this way, the time for saving the distributed time series database can be saved, and the efficiency of saving can be further improved.

[0112] In some embodiments, the above S103, wherein the plurality of task consumption node servers generate summary table data according to the processing tasks and save the summary table data to the distributed time series database, further includes:

[0113] The task consumption node server saves the feature point data according to the device name, the level of the feature point data and the time partition key as the primary key. Among them, the expression of the time partition key is:

[0114] pk=dataTime / (MaxCapacity / EachItemSpace*n*Freq*2 (R-1) )

[0115] Among them, pk is the time partition key, dataTime is the time of the feature point data, MaxCapacity is the preset maximum capacity to ensure query performance, EachItemSpace is the space occupied by the preset device when calculating all aggregation functions, n is the second preset data volume, Freq is the data collection frequency of the device, and R is the level of the feature point data.

[0116] In an embodiment of the present application, the distributed time series database may be a Cassandra database, which allows combining multiple columns of data in the database as a primary key. All information with the same primary key will be written to the same data block. In this way, when retrieving data, you only need to determine the position of the primary key before you can directly read all the data of the primary key. However, if too much data is inserted into the same primary key, it will have a great impact on device performance. Therefore, a reasonable formula is needed to store an appropriate amount of data in each partition.

[0117] For example, in some application scenarios, if it is found through testing that the data stored under each primary key (partition key) does not exceed 10Mb, the subsequent query performance of the device can be guaranteed.

[0118] At the same time, since the storage space size of the data structure of each feature point mainly depends on the number of parameters of the device and the number of types of aggregation functions that need to be calculated for each parameter, the following expression should be satisfied when determining the space occupied by the data:

[0119] EachItemSpace=ParamtearCountmax*FuncCountmax*EachFuncSpace

[0120] That is, the space occupied by each piece of data, EachItemSpace, is equal to the maximum number of parameters, ParamtearCountmax, multiplied by the total number of functions, FuncCountmax, multiplied by the space occupied by each function result, EachFuncSpace.

[0121] According to the above expression, in the application scenario of this embodiment, the maximum space occupied by a single feature point data of a single device is 140 KB.

[0122] In an embodiment of the present application, in order to meet the requirements of device space occupation, the feature point data of the same device and the same level within a continuous time period can be placed in the same partition (or data block). Therefore, the primary key can be set to include: device name, level of feature point data and time partition key, where the expression of the time partition key is as shown in the above formula. In this way, not only the requirements of device space occupation are met, but also the query server can quickly locate the location of the feature point data in the distributed time series database according to the primary key, thereby improving the efficiency of aggregate query.

[0123] In some embodiments, Figure 8 A schematic diagram of the structure of a query server provided by an embodiment of the present invention, such as Figure 8 As shown, the query server may include: a parameter conversion module and a query module.

[0124] Fig. 9 A flow chart of a method for determining a query result by a query server provided in an embodiment of the present invention, such as Fig. 9 As shown, the above S104 may specifically include the following steps S1041-S1045:

[0125] S1041. The query server determines a query time span according to the total query start time and the total query end time.

[0126] In one implementation, this step can be executed by the parameter conversion module in the query server. Specifically, the parameter conversion module can determine the query time span according to the total query start time and the total query end time. The query time span is the difference between the total query end time and the total query start time, that is, the query time span satisfies the following expression:

[0127] timespan = endTime – startTime;

[0128] where timespan is the query time span, endTime is the total query end time, and startTime is the total query start time.

[0129] S1042. The query server determines the basic time window according to the data acquisition frequency of the device corresponding to the query device name and the second preset data volume.

[0130] In one implementation, this step can be executed by the parameter conversion module in the query server. Specifically, the parameter conversion module can determine the basic time window according to the data acquisition frequency of the device corresponding to the query device name and the second preset data volume. Among them, the basic time window is the product of the data acquisition frequency of the device corresponding to the query device name and the second preset data volume, that is, the basic time window satisfies the following expression:

[0131] window = n * Freq;

[0132] where window is the basic time window, n is the second preset data volume, and Freq is the data acquisition frequency of the device.

[0133] S1043. The query server determines the query level of the feature point data according to the query time span and the basic time window.

[0134] In one implementation, this step can be executed by the parameter conversion module in the query server. Specifically, the parameter conversion module can determine the query level of the feature point data according to the above query time span and the above basic time window. Among them, the query level of the feature point data satisfies the following expression:

[0135] When X * n * window < timespan ≤ (X + 1) * n * window, the query level of the feature point data is X + 1, where timespan is the query time span, window is the basic time window, and n is the second preset data volume.

[0136] Exemplarily, when window < timespan ≤ n * window, the query level of the feature point data is 1. When n * window < timespan ≤ 2 * n * window, the query level of the feature point data is 2. When 2 * n * window < timespan ≤ 3 * n * window, the query level of the feature point data is 3.

[0137] S1044. The query server determines the time partition key query range according to the total query start time, the total query end time, the basic time window, and the query level of the feature point data.

[0138] In one implementation, this step can be executed by the parameter conversion module in the query server. Specifically, the parameter conversion module can determine the time partition key query range according to the following expression:

[0139] pk start = QuerystartTime / (MaxCapacity / EachItemSpace * window * 2 (l-1) )

[0140] ~pk end = QueryendTime / (MaxCapacity / EachItemSpace * window * 2 (l-1) );

[0141] where pk start is the start range of the time partition key query, pk end is the end range of the time partition key query, l is the query level of the feature point data, QuerystartTime is the total query start time, QueryendTime is the total query end time; MaxCapacity is the preset maximum capacity to ensure query performance, EachItemSpace is the preset occupied space size of the device when calculating all aggregation functions, and both MaxCapacity and EachItemSpace are fixed values.

[0142] S1045. The query server determines and returns the query result in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time, and the total query end time.

[0143] In one implementation, this step may be performed by a query module in a query server. Specifically, the query module may determine and return query results in a distributed time series database based on the converted query conditions, namely, the query device name, the query level of the feature point data, the time partition key range, the total query start time, and the total query end time.

[0144] In some embodiments, in the above S1045, the query server determines and returns the query result in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time and the total query end time, which may specifically include the following steps S10451-S10459:

[0145] S10451. The query server obtains the first feature point data result by querying in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time and the total query end time. The first feature point data result is arranged in ascending order according to the start time.

[0146] First, the query server can query the distributed time series database according to the above query conditions to obtain the first feature point data results, and arrange the first feature point data results in time series, that is, the first feature point data results can be arranged in ascending order according to the starting time.

[0147] S10452. The query server determines whether the smallest start time in the first feature point data result is equal to the total query start time, and whether the largest end time in the first feature point data result is equal to the total query end time.

[0148] Further, the query server can determine whether the time span of the first feature point data result is the same as the query time span (ie, the difference between the total query end time and the total query start time). Specifically, there are four situations between the time span of the first feature point data result and the query time span.

[0149] Fig.10 A schematic diagram of the time span of the first feature point data result provided by an embodiment of the present invention. In the first case, Fig.10 As shown in (a), the total time span of all first feature point data results completely covers the query time span, that is, the smallest start time in the first feature point data results is equal to the total query start time, and the largest end time in the first feature point data results is equal to the total query end time.

[0150] The second case, such as Fig.10As shown in (b), both ends of the total time span of all first feature point data results do not cover the query time span, that is, the smallest start time in the first feature point data results is not equal to the total query start time, and the largest end time in the first feature point data results is not equal to the total query end time.

[0151] The third case, such as Fig.10 As shown in (c), the starting end of the total time span of all first feature point data results is the same as the starting end of the query time span, but the ending end is different. That is, the smallest starting time in the first feature point data results is equal to the total query starting time, and the largest ending time in the first feature point data results is not equal to the total query ending time.

[0152] The fourth case, such as Fig.10 As shown in (d), the end of the total time span of all first feature point data results is the same as the end of the query time span, but the start is different. That is, the smallest start time in the first feature point data results is not equal to the total query start time, and the largest end time in the first feature point data results is equal to the total query end time.

[0153] S10453. For the first case mentioned above, that is, when the smallest start time in the first feature point data result is equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, the query server uses the first feature point data result as the query result.

[0154] S10454. For the second situation mentioned above, the query server finds that the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and the query level of the feature point data is greater than 1.

[0155] The query server uses the total query start time as the first sub-query start time, the minimum start time as the first sub-query end time, and the query level of the feature point data minus 1 as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the second feature point data result. The maximum end time is used as the second sub-query start time, the total query end time is used as the second sub-query end time, and the query level of the feature point data minus 1 is used as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the third feature point data result. This continues until the minimum start time in the second feature point data result is equal to the total query start time, and the maximum end time in the third feature point data result is equal to the total query end time. The first feature point data result, the second feature point data result, and the third feature point data result are used as query results.

[0156] S10455. For the third case above, the query server finds that the smallest start time in the first feature point data result is equal to the total query start time, the largest end time in the first feature point data result is not equal to the total query end time, and the query level of the feature point data is greater than 1.

[0157] The query server uses the maximum end time as the start time of the third sub-query, the total query end time as the end time of the third sub-query, and the query level of the feature point data minus 1 as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the fourth feature point data result, until the maximum end time in the fourth feature point data result is equal to the total query end time, and the first feature point data result and the fourth feature point data result are used as the query results.

[0158] S10456. For the fourth situation mentioned above, the query server finds that the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, and the query level of the feature point data is greater than 1.

[0159] The query server uses the total query start time as the fourth sub-query start time, the minimum start time as the fourth sub-query end time, and the query level of the feature point data minus 1 as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the fifth feature point data result, until the minimum start time in the fifth feature point data result is equal to the total query start time, and the first feature point data result and the fifth feature point data result are used as the query results.

[0160] S10457. For the second situation mentioned above, the query server finds that the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and the query level of the feature point data is equal to 1.

[0161] The query server uses the total query start time as the fifth query start time, the minimum start time as the fifth query end time, queries the original time series data to obtain the first query time series data, and calculates the sixth feature point data result based on the first query time series data and the aggregation function. Use the maximum end time as the sixth query start time, the total query end time as the sixth query end time, query the original time series data to obtain the second query time series data, and calculate the seventh feature point data result based on the second query time series data and the aggregation function. The first feature point data result, the sixth feature point data result, and the seventh feature point data result are used as query results.

[0162] S10458. For the third situation mentioned above, the query server finds that the smallest start time in the first feature point data result is equal to the total query start time, the largest end time in the first feature point data result is not equal to the total query end time, and the query level of the feature point data is equal to 1.

[0163] The query server uses the maximum end time as the start time of the seventh query and the total query end time as the end time of the seventh query, queries the original time series data to obtain the third query time series data, and calculates the eighth feature point data result based on the third query time series data and the aggregation function. The first feature point data result and the eighth feature point data result are used as the query result.

[0164] S10459. For the fourth situation mentioned above, the query server finds that the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, and the query level of the feature point data is equal to 1.

[0165] The query server uses the total query start time as the eighth query start time, and the minimum start time as the eighth query end time, queries the original time series data to obtain the fourth query time series data, and calculates the ninth feature point data result based on the fourth query time series data and the aggregation function. The first feature point data result and the ninth feature point data result are used as the query result.

[0166] In some embodiments, as described above Figure 7 As shown, the distributed aggregation query system provided in the embodiment of the present application may also include: a distributed configuration center server. Then the method also includes:

[0167] The distributed configuration center server sends a configuration file to the task consumption node server. The configuration file is used to indicate the aggregation function of the calculation.

[0168] The task consumption node server receives the configuration file and calculates the aggregation function according to the configuration file to obtain the summary table data.

[0169] In the embodiment of the present application, the distributed configuration center technology is adopted to configure some special aggregate functions that need to be calculated or aggregate functions that are excluded from calculation in the distributed configuration center server in the form of configuration files. The instant notification mechanism of the distributed configuration center server ensures that all task consumption node servers can instantly and synchronously obtain the aggregate functions that should be additionally calculated or excluded from calculation for a certain parameter of a certain device.

[0170] Specifically, the distributed configuration center server may store multiple preset configuration files, each of which may correspond to one or more aggregation functions. In addition, the distributed configuration center server may also provide an interactive interface for users to modify the configuration files online, and add or delete aggregation functions.

[0171] After startup, the task consumption node server can use the long polling mechanism to send an HTTP request to the distributed configuration center server. The request parameters of the HTTP request carry a wait parameter, which means that during the wait time interval, if the configuration file of the distributed configuration center server changes, the HTTP request can be returned immediately, and the return value carries the latest configuration file. If the wait time interval is exceeded, it will be returned actively. After receiving any kind of return, the task consumption node server will send the same HTTP request again. In this way, it can be ensured that after the configuration of the distributed configuration center server is updated, the task consumption node server can immediately receive the relevant configuration file, thereby ensuring that the task consumption node server can accurately determine the corresponding aggregation information according to the aggregation function and generate summary table data.

[0172] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0173] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0174] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0176] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A distributed summary table data generation and query method, characterized in that: Applied to a distributed aggregate query system, the distributed aggregate query system comprises: a task generation node server, a plurality of task consumption node servers and a query server; the method comprises: The task generation node server receives original time series data, and the original time series data includes: a device name; The task generation node server creates a processing task according to the device name and the acquisition time of the original time series data, wherein the processing task is used to instruct the task consumption node server to generate summary table data; The plurality of task consumption node servers generate summary table data according to the processing tasks, and save the summary table data into a distributed time series database; The query server converts the received query parameters into query conditions, and determines and returns query results in the distributed time series database according to the query conditions; The task generation node server creates a processing task according to the device name and the acquisition time of the original time series data, including: The task generation node server monitors the received original time series data and extracts the device name; The task generation node server determines a task generation time corresponding to a device according to the device name, wherein the task generation time is determined according to a data collection frequency of the device and a first preset data volume, and the device corresponds to the device name; The task generation node server creates a processing task corresponding to the device when the collection time of the original time series data meets the task generation time; The multiple task consumption node servers generate summary table data according to the processing tasks, and save the summary table data into a distributed time series database, including: The task consumption node server queries the task information of the processing task, where the task information of the processing task includes: task start time, task end time and device name; The task consumption node server applies for a first distributed lock according to the task information of the processing task, where the first distributed lock is used to prevent the processing task from being repeatedly processed in the same period; When the task consumption node server applies for the first distributed lock, the task consumption node server generates corresponding summary table data according to the processing task; Wherein, when the task consumption node server applies for the first distributed lock, it generates corresponding summary table data according to the processing task, including: The task consumption node server obtains the historical calculation result corresponding to the device according to the device name; The task consumption node server determines the original time series data corresponding to the device according to the task start time and the task end time; The task consumption node server generates summary table data according to the original time series data corresponding to the device and the historical calculation results; The summary table data includes: L levels of feature point data, the L levels of feature point data include 2 (L-1) * at least one aggregation information of n pieces of the original time series data, wherein L is a positive integer greater than or equal to 1, and n is a second preset data amount; The original time series data also includes: at least one parameter and a value corresponding to the parameter; The task consumption node server generates summary table data according to the original time series data corresponding to the device and the historical calculation results, including: The task consumption node server obtains L levels of data structures corresponding to the device, the L levels of data structures correspond to the L levels of feature point data one by one, and the data structures include: calculation start time, calculation end time and aggregation function; The task consumption node server sorts the L levels of data structures in the order of the calculation end time from small to large, and in the case where the calculation end time is the same, sorts the L levels of data structures in the order of the calculation start time from small to large, to obtain the sorted L levels of data structures; The task consumption node server calculates the aggregation function in each level of the data structure of the L levels according to the value corresponding to the parameter in the original time series data and the historical calculation result, and in the order of the sorted L levels of data structures, to obtain the corresponding feature point data of the L levels; When the calculation end time of the data structure is greater than the collection time of the original time series data, the task consumption node server saves the feature point data of the level corresponding to the calculation end time.

2. The method according to claim 1, characterized in that: When the task consumption node server applies for the first distributed lock, after generating corresponding summary table data according to the processing task, the method further includes: The task consumption node server applies for a second distributed lock; When the task consumption node server applies for the second distributed lock and the second distributed lock is the same as the first distributed lock, the task consumption node server saves the summary table data into the distributed time series database.

3. The method according to claim 1, characterized in that The plurality of task consumption node servers generate summary table data according to the processing tasks, and save the summary table data into a distributed time series database, further comprising: The task consumption node server stores the feature point data according to the device name, the level of the feature point data and the time partition key as the primary key; The expression of the time partition key is: pk=dataTime / (MaxCapacity / EachItemSpace*n*Freq*2 (R-1) ); Among them, pk is the time partition key, dataTime is the time of the feature point data, MaxCapacity is the preset maximum capacity to ensure query performance, EachItemSpace is the space occupied by the preset device when calculating all aggregation functions, n is the second preset data volume, Freq is the data collection frequency of the device, and R is the level of the feature point data.

4. The method according to claim 1, characterized in that: The distributed aggregate query system further includes: a distributed configuration center server; the method further includes: The distributed configuration center server sends a configuration file to the task consumption node server, where the configuration file is used to indicate the calculated aggregation function; The task consumption node server receives the configuration file, and calculates the aggregation function according to the configuration file to obtain the summary table data.

5. The method according to claim 1, characterized in that The query parameters include: total query start time, total query end time and query device name; The query server converts the received query parameters into query conditions, and determines and returns query results in the distributed time series database according to the query conditions, including: The query server determines a query time span according to the total query start time and the total query end time, and the query time span is the difference between the total query end time and the total query start time; The query server determines a basic time window according to the data acquisition frequency of the device corresponding to the query device name and the second preset data volume, and the basic time window is the product of the data acquisition frequency of the device corresponding to the query device name and the second preset data volume; The query server determines the query level of the feature point data according to the query time span and the basic time window, and the query level of the feature point data satisfies the following expression: When X*n*window < timespan ≤ (X+1)*n*window, the query level of the feature point data is X+1, where timespan is the query time span, window is the basic time window, and n is the second preset data volume; The query server determines a time partition key query range according to the total query start time, the total query end time, the basic time window, and the query level of the feature point data, and the time partition key query range satisfies the following expression: pk start ~pk end ; Among them, pk start = QuerystartTime / (MaxCapacity / EachItemSpace * window * 2 (l-1) ); pk end =QueryendTime / (MaxCapacity / EachItemSpace*window*2 (l-1) ); Among them, pk start The starting range of the time partition key query, pk end is the end range of the time partition key query, l is the query level of the feature point data, QuerystartTime is the total query start time, QueryendTime is the total query end time, MaxCapacity is the preset maximum capacity to ensure query performance, and EachItemSpace is the space occupied by the preset device when calculating all aggregation functions; The query server determines and returns a query result in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time, and the total query end time.

6. The method according to claim 5, characterized in that The query server determines and returns a query result in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time, and the total query end time, including: The query server queries a first feature point data result in the distributed time series database according to the query device name, the query level of the feature point data, the time partition key range, the total query start time, and the total query end time, and the first feature point data result is arranged in ascending order of the start time; The query server determines whether the smallest start time in the first feature point data result is equal to the total query start time, and whether the largest end time in the first feature point data result is equal to the total query end time; When the smallest start time in the first feature point data result is equal to the total query start time and the largest end time in the first feature point data result is equal to the total query end time, the query server uses the first feature point data result as the query result; The query server, when the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and when the query level of the feature point data is greater than 1, uses the total query start time as the first sub-query start time, uses the smallest start time as the first sub-query end time, and uses the query level of the feature point data minus 1 as the first sub-query level of the feature point data to query again in the distributed time series database to obtain a second feature point data result; uses the largest end time as the second sub-query start time, uses the total query end time as the second sub-query end time, and uses the query level of the feature point data minus 1 as the first sub-query level of the feature point data to query again in the distributed time series database to obtain a third feature point data result; until the smallest start time in the second feature point data result is equal to the total query start time, and the largest end time in the third feature point data result is equal to the total query end time, the first feature point data result, the second feature point data result, and the third feature point data result are used as the query result; The query server finds that the smallest start time in the first feature point data result is equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and when the query level of the feature point data is greater than 1, the largest end time is used as the third sub-query start time, the total query end time is used as the third sub-query end time, and the query level of the feature point data minus 1 is used as the first sub-query level of the feature point data to query again in the distributed time series database to obtain a fourth feature point data result, until the largest end time in the fourth feature point data result is equal to the total query end time, and the first feature point data result and the fourth feature point data result are used as the query result; The query server finds that the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, and when the query level of the feature point data is greater than 1, the total query start time is used as the fourth sub-query start time, the smallest start time is used as the fourth sub-query end time, and the query level of the feature point data minus 1 is used as the first sub-query level of the feature point data to query again in the distributed time series database to obtain the fifth feature point data result, until the smallest start time in the fifth feature point data result is equal to the total query start time, and the first feature point data result and the fifth feature point data result are used as the query results; The query server, when the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and when the query level of the feature point data is equal to 1, uses the total query start time as the fifth sub-query start time, and uses the smallest start time as the fifth sub-query end time, queries the original time series data to obtain the first query time series data, and calculates the sixth feature point data result according to the first query time series data and the aggregation function; uses the largest end time as the sixth sub-query start time, and uses the total query end time as the sixth sub-query end time, queries the original time series data to obtain the second query time series data, and calculates the seventh feature point data result according to the second query time series data and the aggregation function; uses the first feature point data result, the sixth feature point data result, and the seventh feature point data result as the query result; The query server, when the smallest start time in the first feature point data result is equal to the total query start time, and the largest end time in the first feature point data result is not equal to the total query end time, and when the query level of the feature point data is equal to 1, uses the largest end time as the seventh minute query start time, and uses the total query end time as the seventh minute query end time, queries the original time series data to obtain the third query time series data, and calculates the eighth feature point data result based on the third query time series data and the aggregation function; uses the first feature point data result and the eighth feature point data result as the query result; The query server finds that the smallest start time in the first feature point data result is not equal to the total query start time, and the largest end time in the first feature point data result is equal to the total query end time, and when the query level of the feature point data is equal to 1, the total query start time is used as the eighth query start time, and the smallest start time is used as the eighth query end time, the original time series data is queried to obtain the fourth query time series data, and the ninth feature point data result is calculated based on the fourth query time series data and the aggregation function; the first feature point data result and the ninth feature point data result are used as the query results.