Time series data pre-calculation method, device and equipment

Abstract

The invention relates to a time series data pre-calculation method, device and equipment. The method comprises the following steps: receiving a data calling request sent by a time sequence database; obtaining application data through an extraction interface according to the data calling request; searching a target rule matched with the application data; pre-calculating the application data according to the target rule to obtain a calculation result in a preset data format; and calling a target library function, and sending the calculation result of the preset data format to the time sequence database. Therefore, a part of pre-calculation process can be sunk to an acquisition end, the calculation pressure of a time sequence database is reduced, and the performance loss caused by large-scale time sequence database pre-calculation is reduced; and the resource utilization rate of the whole monitoring system is reduced, the query performance of the database is improved, and the resource loss of the acquisition end is reduced under the condition that the original data and the pre-calculated data are reported at the same time.

Description

Technical Field

[0001] This application relates to the field of big data computing technology, and particularly to a method and apparatus for pre-computing time-series data, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] During the monitoring and use of large-scale complex distributed systems, it is generally necessary to observe, analyze, and alarm many monitoring indicators. Therefore, it is necessary to perform secondary calculations on the collected original monitoring data for deeper analysis later. Time-series databases are widely used in the field of application monitoring due to their characteristics of high throughput writing ability, high aggregation, and multi-dimensional query ability.

[0003] In traditional technologies, the pre-computation of original monitoring data is usually completed by a time-series database, that is, a series of secondary calculations are performed on the collected original monitoring data based on the pre-computation module in the time-series database, and then the calculated data is reported.

[0004] However, the above pre-computation method will consume the storage and query resources of the time-series database. Especially in large-scale monitoring systems, it will seriously affect the performance of the time-series database. Summary of the Invention

[0005] Based on this, it is necessary to provide a method and apparatus for pre-computing time-series data, a computer device, a computer-readable storage medium, and a computer program product that can reduce the performance loss of pre-computation on the time-series database and improve the query performance of the time-series database for the above technical problems.

[0006] In a first aspect, this application provides a method for pre-computing time-series data, including:

[0007] Receiving a data call request sent by a time-series database;

[0008] According to the data call request, obtaining application data through an extraction interface;

[0009] Searching for a target rule that matches the application data;

[0010] According to the target rule, performing pre-computation on the application data to obtain a calculation result in a preset data format;

[0011] Invoking a target library function to send the calculation result in the preset data format to the time-series database.

[0012] In one of the embodiments, the obtaining application data through an extraction interface according to the data call request includes:

[0013] Determine the input parameters and return values corresponding to the application data according to the data call request;

[0014] Determine the returned data format according to the input parameters and the return values; wherein, the returned data format includes at least one of: metric name, label name, label value, data value, data timestamp, time series data type, discard flag;

[0015] Obtain the application data according to the returned data format through the extraction interface.

[0016] In one embodiment, the finding of the target rule matching the application data includes:

[0017] Traverse the label matching index table according to the metric name corresponding to the application data;

[0018] Find the matching instance list corresponding to the application data in the label matching index table; the matching instance list is used to store matching instances, and the matching instance includes: label name, matching symbol, and matching value;

[0019] Traverse the matching instance list according to the label value corresponding to the application data;

[0020] If there is a matching instance corresponding to the application data, obtain the rule ID list corresponding to the application data from the label rule index table;

[0021] Take the union of the rule ID lists corresponding to each label value of the application data to obtain the target rule ID list;

[0022] Perform a bitwise AND operation on the target rule ID list to obtain the target rule ID;

[0023] Find the rule configuration corresponding to the target rule ID from the rule ID index table to obtain the target rule matching the application data.

[0024] In one embodiment, the pre-computing the application data according to the target rule includes:

[0025] In the case where the target rule indicates that the pre-computation type is a single-sequence type, perform at least one of the following pre-computations on the application data:

[0026] Calculate the average rate within a specified time range;

[0027] Calculate the minimum value within a specified time range;

[0028] Calculate the maximum value within a specified time range;

[0029] Calculate the average value within a specified time range;

[0030] Among them, the calculation results of the single - sequence type are stored in accordance with the agreed format, or merged and stored in the application data.

[0031] In one embodiment, the pre - calculating the application data according to the target rule includes:

[0032] When the target rule indicates that the pre - calculation type is an aggregation type, perform at least one of the following pre - calculations on the application data:

[0033] Calculate the sum of all sequence values;

[0034] Calculate the minimum value of all sequence values;

[0035] Calculate the maximum value of all sequence values;

[0036] Calculate the average value of all sequences;

[0037] Among them, the calculation results of the aggregation type only retain the tags and values specified by the output tag fields in the tag name and tag value.

[0038] In one embodiment, before looking up the target rule that matches the application data, the method further includes:

[0039] Generate rule configurations according to the configuration file;

[0040] Construct matching instances according to the rule configurations;

[0041] Form an array according to the loading order of all matching instances;

[0042] Construct a tag - matching index table with the tag name as the key and the position list of the matching instance in the array as the value;

[0043] Construct a tag - rule index table with the position of the matching instance in the array as the key and the rule ID list as the value;

[0044] Generate a rule ID and a rule - ID index table in a bit - operation manner according to the position of the matching instance; the rule - ID index table is used to store rule configurations.

[0045] In one embodiment, before pre - calculating the application data according to the target rule, the method further includes:

[0046] Determine the hash value according to the metric name, tag name, and tag value corresponding to the application data;

[0047] Store the data value and data timestamp of the application data in the cache with the hash value as the key;

[0048] Regularly clean up the expired data in the cache.

[0049] In one of the embodiments, the method further includes:

[0050] Parse the calculation result of the preset data format through the time series database, and split out the original metrics and pre-calculated metrics of the application data according to the agreed format.

[0051] In a second aspect, the present application further provides a time series data pre-calculation device, and the device includes:

[0052] A receiving module, configured to receive a data call request sent by a time series database;

[0053] An extraction module, configured to obtain application data through an extraction interface according to the data call request;

[0054] A search module, configured to search for a target rule matching the application data;

[0055] A pre-calculation module, configured to perform pre-calculation on the application data according to the target rule to obtain a calculation result in a preset data format;

[0056] A sending module, configured to call a target library function to send the calculation result in the preset data format to the time series database.

[0057] In a third aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0058] Receive a data call request sent by a time series database;

[0059] Obtain application data through an extraction interface according to the data call request;

[0060] Search for a target rule matching the application data;

[0061] Perform pre-calculation on the application data according to the target rule to obtain a calculation result in a preset data format;

[0062] Call a target library function to send the calculation result in the preset data format to the time series database.

[0063] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0064] Receive a data call request sent by a time series database;

[0065] Obtain application data through an extraction interface according to the data call request;

[0066] Search for a target rule that matches the application data;

[0067] Pre-compute the application data according to the target rule to obtain a calculation result in a preset data format;

[0068] Call a target library function to send the calculation result in the preset data format to the time series database.

[0069] In a fifth aspect, the present application also provides a computer program product, including a computer program, which when executed by a processor implements the following steps:

[0070] Receive a data call request sent by the time series database;

[0071] Obtain application data through an extraction interface according to the data call request;

[0072] Search for a target rule that matches the application data;

[0073] Pre-compute the application data according to the target rule to obtain a calculation result in a preset data format;

[0074] Call a target library function to send the calculation result in the preset data format to the time series database.

[0075] The above time series data pre-computation method, device, computer device, computer-readable storage medium, and computer program product receive a data call request sent by the time series database; according to the data call request, obtain application data through an extraction interface; thus, application data can be obtained through a dedicated extraction interface. Search for a target rule that matches the application data; according to the target rule, pre-compute the application data to obtain a calculation result in a preset data format; thus, part of the pre-computation process can be sunk to the acquisition end, reducing the calculation pressure on the time series database and reducing the performance loss of large-scale time series database pre-computation. Call a target library function to send the calculation result in the preset data format to the time series database. Thus, the resource utilization rate of the overall monitoring system can be reduced, the query performance of the database can be improved, and the resource loss at the acquisition end can be reduced when reporting both raw data and pre-computed data at the same time. Description of the Drawings

[0076] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the accompanying drawings required for the description of the embodiments of the present application or related technologies. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related accompanying drawings can also be obtained based on these drawings.

[0077] Figure 1 Schematic diagram of the application architecture of the pre-computation method for time-series data in an embodiment Figure 1 ;

[0078] Figure 2 Schematic diagram of the application architecture of the pre-computation method for time-series data in an embodiment Figure 2 ;

[0079] Figure 3 Schematic flow chart of the pre-computation method for time-series data in an embodiment;

[0080] Figure 4 Overall schematic flow chart of the pre-computation method for time-series data in an embodiment;

[0081] Figure 5 Schematic flow chart of the pre-computation method for time-series data in another embodiment;

[0082] Figure 6 Schematic flow chart of the process of constructing pre-computation rules in an embodiment;

[0083] Figure 7 Schematic flow chart of the process of matching pre-computation rules in an embodiment;

[0084] Figure 8 Schematic flow chart of the pre-computation method for time-series data in yet another embodiment;

[0085] Figure 9 Schematic diagram of the application architecture of the pre-computation method for time-series data in an embodiment Figure 3 ;

[0086] Figure 10 Schematic diagram of the implementation principle of the acquisition end in an embodiment;

[0087] Figure 11 Block diagram of the structure of the pre-computation device for time-series data in an embodiment;

[0088] Figure 12 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0089] To make the objectives, technical solutions and advantages of this application clearer, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining this application and are not used to limit this application.

[0090] To facilitate the understanding of the solutions in each embodiment of this application, the following briefly explains the professional terms that may appear in each embodiment of this application:

[0091] Content Delivery Network (CDN), whose main function is to distribute content to servers closer to users, thereby optimizing the user experience. Through this method, users can obtain information faster, greatly reducing the waiting time. At the same time, CDN can also protect the data security of users.

[0092] The pre-computation method for time series data provided by the embodiments of this application can be applied to an Figure 1 application architecture as shown. The time series database cluster can dock with application data through a customized collector (Exporter), and convert the application data into a standard collection format for reporting during collection. In CDN monitoring, the reported raw data cannot be directly provided for query externally. One reason is that calculations need to be performed, such as calculating the rate and average within 5 minutes; another reason is that the volume of raw data is huge, affecting the performance of the time series database, and data needs to be aggregated according to the business to reduce the query volume; the third reason is that the same calculation is performed for each query, wasting resources. Therefore, it is necessary to pre-compute the data after collecting the data.

[0093] Exemplarily, as Figure 2 shown, it provides a general implementation solution for a time series database, that is, the time series database communicates with the Exporter through a built-in collection module, and then uses the Exporter to dock with application data; the application data obtained by the collection module is processed through a built-in pre-computation module and stored through a storage module, and finally query services are provided through a query module. The disadvantages of this solution: one is that the pre-computation module executes calculation tasks at regular intervals, resulting in a certain delay; the other is that the calculation tasks will consume the storage and query resources of the database, which is particularly serious in large-scale monitoring systems.

[0094] Aiming at the problems existing in the existing time series data calculation methods, the embodiments of this application aim to sink part of the pre-computation process to the collector side, and let the collector side perform part of the calculation and aggregation first to relieve the calculation pressure on the time series database. In addition, the collector side can also improve the performance and reduce the resource usage rate of the overall monitoring system by means of caching and agreeing on the data reporting format.

[0095] In an exemplary embodiment, as Figure 3As shown, a method for pre-computing time-series data is provided. When this method is applied to the collection end, it includes the following steps 301 to 305. Among them:

[0096] Step 301: Receive a data call request sent by the time-series database.

[0097] In the embodiments of this application, it is applied to the collection end. The collection end can be an Exporter, or a certain collector or collection module built into the Exporter, which is used to dock application data. Among them, the time-series database end can initiate a data call request to the collection end to instruct the collection end to execute the collection task of application data.

[0098] Step 302: Obtain application data through the extraction interface according to the data call request.

[0099] In the embodiments of this application, the extraction interface (which can be the Fetch interface) can obtain application data according to the data call request and return the application data in accordance with the agreed data format.

[0100] Exemplarily, according to the data call request, determine the input parameters and return values corresponding to the application data; according to the input parameters and return values, determine the returned data format; among them, the returned data format includes at least one of: metric name (metric_name), label name (label_names), label value (label_values), data value (value), data timestamp (timestamp), time-series data type (value_type), discard flag (drop); where drop is applicable to the scenario where the metric will no longer be used after pre-computation. Through the extraction interface, obtain the application data in accordance with the returned data format.

[0101] Step 303: Search for a target rule that matches the application data.

[0102] In the embodiments of this application, after obtaining the application data, it is also necessary to find a target rule that matches it. The target rule is used to indicate the calculation method of the application data. By setting the pre-computation rule for searching for the matching application data and expanding the format for reporting pre-computed data, the memory consumption of the Exporter can be minimized.

[0103] Exemplarily, according to the metric name corresponding to the application data, traverse the label matching index table; search for the matching instance list corresponding to the application data in the label matching index table; the shown matching instance list is used to store matching instances, and the matching instances include: label name, matching symbol, and matching value; according to the label value corresponding to the application data, traverse the matching instance list; if there is a matching instance corresponding to the application data, obtain the rule ID list corresponding to the application data from the label rule index table; take the union of the rule ID lists corresponding to each label value of the application data to obtain the target rule ID list; perform a bitwise AND operation on the target rule ID list to obtain the target rule ID; search for the rule configuration corresponding to the target rule ID in the rule ID index table to obtain the target rule matched by the application data.

[0104] Step 304, according to the target rule, perform pre-calculation on the application data to obtain the calculation result in the preset data format.

[0105] Exemplarily, in the case where the target rule indicates that the pre-calculation type is the single-sequence type, the following operations can be performed on the application data: calculating the average rate within a specified time range, calculating the minimum value within a specified time range, calculating the maximum value within a specified time range, calculating at least one of the average values within a specified time range; wherein, the calculation result of the single-sequence type is stored in accordance with the agreed format, or merged and stored in the application data.

[0106] In the embodiments of the present application, the single-sequence type refers to calculating using the data of the sequence itself within a certain time range, and this time range is specified by the range field in the configuration. Among them, rate: is used to calculate the average rate within a specified time range; min_over_time: is used to calculate the minimum value within a specified time range; max_over_time: is used to calculate the maximum value within a specified time range; vg_over_time: is used to calculate the average value within a specified time range.

[0107] Exemplarily, in the case where the target rule indicates that the pre-calculation type is the aggregation type, perform at least one of the operations of calculating the sum of all sequence values, calculating the minimum value of all sequence values, calculating the maximum value of all sequence values, calculating the average value of all sequences on the application data; wherein, the calculation result of the aggregation type only retains the labels and values specified by the output label field in the label name and label value.

[0108] In the embodiments of the present application, the aggregation type refers to a calculation involving multiple sequences and requires classification through the labels specified by output. Among them, sum: is used to calculate the sum of all sequence values; min: is used to calculate the minimum value of all sequence values; max: is used to calculate the maximum value of all sequence values; avg: is used to calculate the average value of all sequences.

[0109] Among them, the calculation result generates the data format specified by the Fetch interface according to the rule configuration. For the pre-calculated data of the aggregation type, only the labels and values specified by the output field are retained for the label names (label_names) and label values (label_values). For the preservation method of the pre-calculation result of the single-series type, a special format convention needs to be made. If the original data needs to be retained, the calculation result is merged into the original data, that is, two labels are added to the label list of the original data. Among them, the label _rule_name_ = "pre-calculated name field,..." is used to record the metric name after pre-calculation, and the label _rule_value_ = "pre-calculated value,..." is used to record the value of the metric after pre-calculation, separated by multiple empty commas. In this way, no matter how many single-series type pre-calculations are performed, the amount of reported data will not double, thus achieving the purpose of saving memory.

[0110] Step 305, call the target library function to send the calculation result in the preset data format to the time series database.

[0111] In the embodiment of the present application, the target library function can be the prometheus library function, and the calculation result in the preset data format is sent to the time series database by calling the prometheus library function.

[0112] Exemplarily, as Figure 4 shown, for the sake of clarity, the application data without pre-calculation is called the original data. First, traverse the original data and match it with the original data according to the configuration rules; if not matched, save the data; if matched, calculate the hash value according to the metric name (metric_name), label name (label_names), and label value (label_values); then traverse all pre-calculation rules to determine the calculation type; if it is the aggregation type, find and create an aggregator table according to the pre-calculation function (func) field, perform a hash calculation on the values corresponding to the label names specified by the configured output field, find and create the aggregator corresponding to this hash value, and add it to the aggregator table; add the data to the corresponding aggregator and determine whether the pre-calculation traversal is completed; if it is the single-series type, obtain the cache according to the hash value, call the function specified by func for calculation, save the single-series calculation data and then determine whether the pre-calculation traversal is completed; if the pre-calculation traversal is not completed, continue to traverse all pre-calculation rules; if the pre-calculation traversal is completed, determine whether to save the original data; if the original data is saved, traverse the single-series calculation data, merge it into the original data and then save the data; if the original data is not saved, traverse the single-series calculation data and then save the data; determine whether the data traversal is completed. If completed, traverse the aggregator and save the calculation data, and call the prometheus library function to output the metric; if not completed, continue to traverse the original data.

[0113] The above-mentioned pre-computation method for time-series data receives a data call request sent by a time-series database; according to the data call request, application data is obtained through an extraction interface; thus, application data can be obtained through a dedicated extraction interface. A target rule matching the application data is found; according to the target rule, the application data is pre-computed to obtain a computation result in a preset data format; thus, part of the pre-computation process can be sunk to the acquisition end, reducing the computation pressure on the time-series database and reducing the performance loss of large-scale time-series database pre-computation. A target library function is called to send the computation result in the preset data format to the time-series database. Thus, the resource utilization rate of the overall monitoring system can be reduced, the query performance of the database can be improved, and the resource loss at the acquisition end can be reduced when reporting both original data and pre-computed data at the same time.

[0114] In another exemplary embodiment, as Figure 5 shown, a pre-computation method for time-series data is provided. When this method is applied to the acquisition end, it includes the following steps 501 to 506. Among them:

[0115] Step 501: Receive a data call request sent by a time-series database.

[0116] Step 502: According to the data call request, obtain application data through an extraction interface.

[0117] For the specific implementation process and technical effects of steps 501 to 502 in the embodiments of this application, please refer to Figure 3 the relevant descriptions of steps 301 to 302 in the method embodiment shown, which will not be elaborated here.

[0118] Step 503: Build a rule ID and rule ID index table according to the configuration file.

[0119] In the embodiments of this application, pre-computation rules are loaded according to the configuration file to match each piece of data and perform pre-computation. Among them, the configuration file format of the pre-computation rules includes at least one of the following: the metric name (name) of the data after computation, the matching rule (expr), the pre-computation function (func), the output label (output), and the time range (range) of the data required for computation.

[0120] Exemplarily, first generate rule configurations according to a configuration file; construct matching instances based on the rule configurations; form an array according to the loading order of all matching instances; construct a label matching index table with the label name as the key and the position list of the matching instances in the array as the value; construct a label rule index table with the position of the matching instance in the array as the key and the rule ID list as the value; generate a rule ID and a rule ID index table in a bitwise operation manner according to the position of the matching instance; the rule ID index table is used to store the rule configurations.

[0121] Optionally, first construct a rule index according to expr to generate a rule ID. As Figure 6 shown, __name__ corresponds to the metric name (e.g., metric1, metric2, metric3, etc.), host corresponds to the matching symbol, group corresponds to an array, code corresponds to the status code, and state corresponds to the status. Among them, expr is a series of label matching rules (a combination of label name + matching symbol + matching value). The form of label name + matching symbol + matching value can form a matching instance. First, form an array according to the loading order of all matching instances. Then, construct a hash table (i.e., the label matching index table) with the label name as the key and the position list of the matching instances in the array as the value, construct a hash table (i.e., the label rule index table) with the position of the matching instance in the array as the key and the rule ID list as the value. Finally, generate the unique ID of the pre-computed rule in a bitwise operation manner according to the position of the matching instance included in expr, and generate an ID hash table (i.e., the rule ID index table).

[0122] Step 504, look up the target rule that matches the application data from the rule ID index table.

[0123] Optionally, as Figure 7As shown in the figure, first, according to the data format of the acquired application data: metric name (metric_name), label names (label_names), label values (label_values), data value (value), data timestamp (timestamp), time series data type (value_type), and discard flag (drop), traverse the label names. Then, find the corresponding matching instance list in the label matching index table, traverse the matching instance list, and pass in the corresponding data in the label values for matching; if there is a match, obtain the rule ID list from the label rule index table, and perform a bitwise operation based on the position of this matching instance. Finally, a matching ID can be obtained; otherwise, the rule ID list is empty. It should be noted that for a matching instance with label = "", if the original data does not have this label, the matching of this label will be ignored. Therefore, for such a matching instance, it is considered a successful match by default, and deletion is performed only if the match fails. Take the union of the ID lists of each label match to obtain the final matching rule ID list, then traverse this list and perform a bitwise AND operation with the matching ID. Those with a value greater than 0 are the finally matched rule IDs, and then obtain the corresponding rule configuration from the rule ID index table.

[0124] Step 505, according to the target rule, perform pre-computation on the application data to obtain the calculation result in the preset data format.

[0125] Step 506, call the target library function to send the calculation result in the preset data format to the time series database.

[0126] For the specific implementation process and technical effects of steps 504 to 506 in the embodiments of this application, please refer to Figure 3 the relevant descriptions of steps 303 to 305 in the method embodiment shown in the figure, which will not be elaborated here.

[0127] In the embodiments of this application, an index can be established through pre-computation rules, and the pre-computation rules matching the data can be found through data search, which has higher timeliness and fewer matching times.

[0128] In another exemplary embodiment, as Figure 8 shown, a time series data pre-computation method is provided. When this method is applied to the acquisition end, it includes the following steps 801 to 806. Among them:

[0129] Step 801, receive the data call request sent by the time series database.

[0130] Step 802, according to the data call request, obtain the application data through the extraction interface.

[0131] Step 803, find the target rule matching the application data.

[0132] For the specific implementation processes and technical effects of steps 801 to 803 in the embodiments of the present application, please refer to Figure 3 the relevant descriptions of steps 301 to 303 in the method embodiment shown, which will not be elaborated here.

[0133] Step 804: Store at least a part of the application data in the cache.

[0134] Exemplarily, determine a hash value according to the metric name, tag name, and tag value corresponding to the application data; use the hash value as the key to store the data value and data timestamp of the application data in the cache; regularly clean up the expired data in the cache.

[0135] In the embodiments of the present application, a corresponding single-sequence type calculation function can also be provided. When calling the calculation function, update the maximum time range of the sequence cache according to the calculation range. If multiple pre-calculation rules use this sequence, take the maximum time range among them.

[0136] In the embodiments of the present application, delete the expired data according to the time range of the cache sequence. If all the data has expired, delete the cache of this sequence.

[0137] Step 805: Extract the application data from the cache according to the target rule for pre-calculation to obtain a calculation result in a preset data format.

[0138] Step 806: Call the target library function to send the calculation result in the preset data format to the time series database.

[0139] For the specific implementation processes and technical effects of steps 805 to 806 in the embodiments of the present application, please refer to Figure 3 the relevant descriptions of steps 304 to 305 in the method embodiment shown, which will not be elaborated here.

[0140] In the embodiments of the present application, by setting the cache, the calculation pressure of the pre-calculation can be alleviated, and the resource consumption of the acquisition end can be reduced.

[0141] Exemplarily, as Figure 9 shown, another time series database implementation solution is provided. A collector is set in the acquisition end, and a parser is set in the acquisition module of the time series database. Among them, the implementation principle of the collector is as Figure 10 shown. The parser is used to parse the calculation result in the preset data format and split out the original metrics and pre-calculation metrics of the application data according to the agreed format.

[0142] Combined with Figure 10It can be seen that first, the Fetch interface (extraction interface) receives a data call request from the time series database through the Hypertext Transfer Protocol (HTTP) service, and obtains application data according to the data call request; then, it searches for pre-built pre-computation rules (built based on a configuration file) for the application data. If the amount of application data is large, some of the application data can also be stored in the cache; it pre-computes the application data according to the found pre-computation rules, and calls the prometheus library function to send the calculation result to the time series database through the HTTP service.

[0143] Exemplarily, suppose there is business service data on 100,000 edge nodes that needs to be collected. Each node reports 10 metrics such as inflow and outflow traffic, detailed status codes, and request time according to the domain name granularity. Each node accesses 30 domain names. Then the central time series database needs to collect 30 million metrics, and each node collects 300 metrics. The collection period is 1 minute. The provided query metrics include: Queries Per Second (QPS) value and the average request time of each node. First, determine the metrics of the original data (application data without pre-computation) (such as inflow traffic, outflow traffic, the number of requests for each status code, and the request time for each domain name, etc.). Then configure the pre-computation rules. For example: calculate the inflow bandwidth, taking the average change rate within 5 minutes; calculate the outflow bandwidth, taking the average change rate within 5 minutes; calculate the QPS for each status code, taking the average change rate within 5 minutes; calculate the average request time. Further, obtain the application data, return the data through the Fetch interface, and output the data after matching and calculating through the pre-computation rules.

[0144] In the embodiment of this application, each node only needs to calculate the rates of 270 metrics and calculate the average of 30 metrics, which basically does not consume much resources, effectively disperses the pre-computation pressure of the central time series database, and does not have too much impact on the edge nodes. By calling the prometheus library function, the data is converted into the standard collection format to report the metric data, retaining the original data of the traffic, and discarding the original metrics of the number of requests and request time.

[0145] It should be understood that in actual applications, generally the original metrics are not discarded. If the data is reported in the normal way, the data volume of each node will double, consuming memory. In this solution, the pre-computation result is put into the original metrics as the tags _rule_name_ and _rule_value_, effectively saving nearly half of the memory and having better performance. After the metric data is transmitted to the time series database collection module, a parser is called to parse the data and write it into the storage module.

[0146] In the embodiments of the present application, by sinking part of the pre - calculation to the acquisition end, the calculation pressure on the central time - series database is alleviated. Moreover, the data is pre - calculated immediately after acquisition, with higher timeliness. In addition, an index can be established using the pre - calculation rules, and by looking up the data to match the pre - calculation rules, the timeliness is higher and the number of matches is less. On the basis of meeting the standard acquisition format, the pre - calculation result can also be used as a tag and put into the original data. In the scenario where the original data is not discarded, less memory is used and the memory usage of the acquisition end is basically not affected.

[0147] It should be understood that although the steps in the flowcharts involved in the above - mentioned embodiments are sequentially shown according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above - mentioned embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0148] Based on the same inventive concept, the embodiments of the present application also provide a time - series data pre - calculation device for implementing the time - series data pre - calculation method involved above. The implementation solutions provided by this device to solve problems are similar to the implementation solutions recorded in the above - mentioned method. Therefore, the specific limitations in one or more embodiments of the following time - series data pre - calculation device can refer to the limitations on the time - series data pre - calculation method in the above text, and will not be repeated here.

[0149] In an exemplary embodiment, as Figure 11 shown, a time - series data pre - calculation device is provided, including: a receiving module 1101, an extracting module 1102, a searching module 1103, a pre - calculation module 1104, and a sending module 1105, where:

[0150] The receiving module 1101 is configured to receive a data call request sent by the time - series database;

[0151] The extracting module 1102 is configured to obtain application data through an extraction interface according to the data call request;

[0152] The searching module 1103 is configured to search for a target rule that matches the application data;

[0153] The pre - calculation module 1104 is configured to pre - calculate the application data according to the target rule to obtain a calculation result in a preset data format;

[0154] The sending module 1105 is configured to call a target library function and send the calculation result in a preset data format to the time series database.

[0155] Exemplarily, the extraction module 1102 is specifically configured to: determine input parameters and return values corresponding to the application data according to a data call request; determine the returned data format according to the input parameters and the return values; wherein, the returned data format includes at least one of: metric name, label name, label value, data value, data timestamp, time series data type, discard flag; obtain the application data according to the returned data format through an extraction interface.

[0156] Exemplarily, the lookup module 1103 is specifically configured to: traverse a label matching index table according to the metric name corresponding to the application data; find a list of matching instances corresponding to the application data in the label matching index table; the shown list of matching instances is used to store matching instances, and a matching instance includes: label name, matching symbol, and matching value; traverse the list of matching instances according to the label value corresponding to the application data; if there is a matching instance corresponding to the application data, obtain a list of rule IDs corresponding to the application data from a label rule index table; take the union of the lists of rule IDs corresponding to each label value of the application data to obtain a target list of rule IDs; perform a bitwise AND operation on the target list of rule IDs to obtain a target rule ID; find the rule configuration corresponding to the target rule ID from a rule ID index table to obtain the target rule matched by the application data.

[0157] Exemplarily, the pre-computation module 1104 is specifically configured to: when the target rule indicates that the pre-computation type is a single series type, perform at least one of the following pre-computations on the application data:

[0158] Calculate the average rate within a specified time range;

[0159] Calculate the minimum value within a specified time range;

[0160] Calculate the maximum value within a specified time range;

[0161] Calculate the average value within a specified time range;

[0162] Wherein, the calculation result of the single series type is stored in a convention format or merged and stored in the application data.

[0163] Exemplarily, the pre-computation module 1104 is specifically configured to: when the target rule indicates that the pre-computation type is an aggregation type, perform at least one of the following pre-computations on the application data:

[0164] Calculate the sum of all sequence values;

[0165] Calculate the minimum value of all sequence values;

[0166] Calculate the maximum value of all sequence values;

[0167] Calculate the average value of all sequences;

[0168] Among them, only the tags and values specified by the output tag field are retained in the tag name and tag value in the calculation result of the aggregation type.

[0169] Exemplarily, the above device may further include a configuration module 1106, configured to generate rule configurations according to a configuration file; construct matching instances according to the rule configurations; form an array in the loading order of all matching instances; use the tag name as the key and the position list of the matching instances in the array as the value to construct a tag matching index table; use the position of the matching instance in the array as the key and the rule ID list as the value to construct a tag rule index table; generate a rule ID and a rule ID index table in a bitwise operation manner according to the position of the matching instance; the rule ID index table is used to store the rule configurations.

[0170] Exemplarily, the above device may further include a storage module 1107, configured to determine a hash value according to the metric name, tag name, and tag value corresponding to the application data; use the hash value as the key and store the data value and data timestamp of the application data in the cache; periodically clean up the expired data in the cache.

[0171] Exemplarily, the above device may further include: a parsing module 1108, configured to parse the calculation result of a preset data format through a time series database and split out the original metrics and pre-calculated metrics of the application data in accordance with a predefined format.

[0172] It should be understood that the above parsing module 1108 may be provided at the collection end or in the collection module of the time series database.

[0173] Each module in the above time series data pre-calculation device may be implemented in whole or in part by software, hardware, and their combination. The above modules may be embedded in the processor of the computer device in hardware form or independent thereof, or stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0174] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 12As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store application data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for pre-computing time-series data.

[0175] Those skilled in the art can understand that Figure 12 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0176] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0177] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0178] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0180] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0181] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope recorded in the present application.

[0182] The above embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A pre-computation method for time-series data, characterized in that, The method includes: Receiving a data call request sent by a time series database; Obtaining application data through an extraction interface according to the data call request; Searching for a target rule that matches the application data; Pre-computing the application data according to the target rule to obtain a calculation result in a preset data format; Invoking a target library function to send the calculation result in the preset data format to the time series database.

2. The method according to claim 1, wherein The obtaining application data through an extraction interface according to the data call request includes: Determining input parameters and return values corresponding to the application data according to the data call request; Determining a returned data format according to the input parameters and the return values; wherein, the returned data format includes at least one of an index name, a label name, a label value, a data value, a data timestamp, a time series data type, and a discard flag; Obtaining the application data through the extraction interface according to the returned data format.

3. The method according to claim 1, wherein The searching for a target rule that matches the application data includes: Traversing a label matching index table according to the index name corresponding to the application data; Searching for a matching instance list corresponding to the application data in the label matching index table; the matching instance list is used to store matching instances, and the matching instance includes a label name, a matching symbol, and a matching value; Traversing the matching instance list according to the label value corresponding to the application data; If there is a matching instance corresponding to the application data, obtaining a rule ID list corresponding to the application data from a label rule index table; Taking the union of the rule ID lists corresponding to the respective label values of the application data to obtain a target rule ID list; Performing a bitwise AND operation on the target rule ID list to obtain a target rule ID; Searching for a rule configuration corresponding to the target rule ID from a rule ID index table to obtain a target rule that matches the application data.

4. The method according to claim 1, wherein The pre-computing the application data according to the target rule includes: In the case where the target rule indicates that the pre-computation type is a single sequence type, performing at least one of the following pre-computations on the application data: Calculating an average rate within a specified time range; Calculating a minimum value within a specified time range; Calculating a maximum value within a specified time range; Calculating an average value within a specified time range; Wherein, the calculation result of the single sequence type is stored in a conventional format or merged and stored in the application data.

5. The method according to claim 1, characterized in that, The pre-computing the application data according to the target rule includes: In the case where the target rule indicates that the pre-computation type is an aggregation type, performing at least one of the following pre-computations on the application data: Calculating the sum of all sequence values; Calculating the minimum value of all sequence values; Calculating the maximum value of all sequence values; Calculating the average value of all sequences; Wherein, for the calculation result of the aggregation type, only the labels and values specified by the output label fields are retained in the label name and the label value.

6. The method according to any one of claims 1 to 4, characterized in that Before searching for a target rule that matches the application data, the method further includes: Generating a rule configuration according to a configuration file; Constructing a matching instance according to the rule configuration; Forming an array in the loading order of all matching instances; Construct a tag matching index table with the tag name as the key and the list of positions of matching instances in the array as the value; Construct a tag rule index table with the position of the matching instance in the array as the key and the list of rule IDs as the value; Generate a rule ID and a rule ID index table in a bitwise operation manner according to the position of the matching instance; the rule ID index table is used to store rule configurations.

7. The method according to any one of claims 1 to 4, characterized in that, Before pre-computing the application data according to the target rule, the method further includes: Determine a hash value according to the metric name, tag name, and tag value corresponding to the application data; Store the data value and data timestamp of the application data in the cache with the hash value as the key; Regularly clean up expired data in the cache.

8. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Parse the calculation result of the preset data format through the time series database, and split out the original metrics and pre-computed metrics of the application data according to the agreed format.

9. A pre-computation device for time-series data, characterized in that, The device includes: A receiving module for receiving a data call request sent by the time series database; An extraction module for obtaining application data through an extraction interface according to the data call request; A lookup module for looking up a target rule that matches the application data; A pre-computation module for pre-computing the application data according to the target rule to obtain a calculation result in a preset data format; A sending module for calling a target library function to send the calculation result in the preset data format to the time series database.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images