Data cache supplement method, device, equipment and medium

Abstract

The present application provides a data cache supplement method, apparatus, device and medium, the data cache supplement method comprising parsing a user's first query statement to obtain a first indicator fragment; determining whether there is cached data that matches the first query statement; if there is no cached data that matches the first query statement, determining whether there is cached data that matches the first indicator fragment; if there is cached data that matches the first indicator fragment, and the query time period of the cached data does not completely include the query time period of the first query statement, obtaining the original data corresponding to the first query statement, and supplementing the query time period of the missing cached data in the cached data cluster. Since the query time period of the missing cached data can be supplemented in the cached data cluster, the user's new query statement can be made more likely to match the cached data, which can reduce the query pressure on the original data, thereby improving the speed of data query.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data cache replenishment method, device, equipment and medium. Background Art

[0002] Prometheus is widely used for resource monitoring in cloud-native environments. Monitoring data is growing exponentially. As the amount of monitoring data increases, users often encounter slow responses when using query statements to query data. Query data often times out, causing Prometheus service downtime.

[0003] Prometheus provides a method for establishing recording rules to simplify complex queries, but currently only predefined rules are provided, and querying data still takes a long time. Summary of the Invention

[0004] Embodiments of the present application provide a data cache replenishment method, apparatus, device, and medium for improving the speed of data query in Prometheus.

[0005] In a first aspect, an embodiment of the present application provides a data cache replenishment method, the method comprising:

[0006] Parsing the user's first query statement to obtain a first indicator fragment;

[0007] Based on the cached data in the cached data cluster, determining whether there is cached data that matches the first query statement;

[0008] If there is no cached data matching the first query statement, determining whether there is cached data matching the first indicator fragment;

[0009] If there is cached data that matches the first indicator fragment, determining, for the cached data that matches the first indicator fragment, whether the query time period of the cached data completely includes the query time period of the first query statement;

[0010] If the query time period of the cached data does not completely include the query time period of the first query statement, obtaining the original data corresponding to the first query statement, and supplementing the query time period of the missing cached data in the cached data cluster;

[0011] The query time period in which the cached data is missing is a query time period in which the query time period of the first query statement is not within the query time period of the cached data.

[0012] Optionally, parsing the user's first query statement to obtain a first indicator fragment includes:

[0013] Based on the user's first query statement and the existing recording rules in the cache data cluster, filtering the indicators in the first query statement to obtain a filtered first query statement;

[0014] The filtered first query statement is subjected to fragment extraction based on grammatical rules to obtain a first indicator fragment.

[0015] Optionally, after determining whether there is cached data matching the first query statement based on the cached data in the cached data cluster, the method further includes:

[0016] If there is cached data matching the first query statement, determining, for the cached data matching the first query statement, whether a query time period of the cached data completely includes a query time period of the first query statement;

[0017] If the query time period of the cached data does not completely include the query time period of the first query statement, original data corresponding to the first query statement is obtained, and the query time period of the missing cached data is supplemented in the cached data cluster.

[0018] Optionally, after determining whether the query time period of the cached data completely includes the query time period of the first query statement, the method further includes:

[0019] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, then obtaining the cache interface data corresponding to the first query statement; or

[0020] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rule, and the cache indicator fragment reorganization data is obtained.

[0021] Optionally, after determining whether there is cached data matching the first indicator segment, the method further includes:

[0022] If there is no cached data matching the first indicator fragment, original data corresponding to the first query statement is obtained, and the first indicator fragment is supplemented in the cached data cluster.

[0023] Optionally, after determining whether the query time period of the cached data completely includes the query time period of the first query statement, the method further includes:

[0024] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, converting the cached data into cache indicator fragment reconstructed data according to the calculation rule, and obtaining the cache indicator fragment reconstructed data; or

[0025] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rules and calculation rules, and the cache indicator fragment reorganization data is obtained.

[0026] In a second aspect, an embodiment of the present application further provides a data cache supplement device, comprising:

[0027] A parsing unit, configured to parse a first query statement of a user and obtain a first indicator segment;

[0028] a judgment unit, configured to judge, based on cached data in the cached data cluster, whether there is cached data matching the first query statement; if no cached data matching the first query statement is present, then judging whether there is cached data matching the first indicator fragment; if there is cached data matching the first indicator fragment, then judging, for the cached data matching the first indicator fragment, whether a query time period of the cached data completely includes a query time period of the first query statement;

[0029] An acquisition and supplementation unit is used to obtain the original data corresponding to the first query statement if the query time period of the cached data does not completely include the query time period of the first query statement, and supplement the query time period of the missing cached data in the cached data cluster; wherein, the query time period of the missing cached data is a query time period in which the query time period of the first query statement is not within the query time period of the cached data.

[0030] Optionally, the parsing unit is specifically configured to:

[0031] Based on the user's first query statement and the existing recording rules in the cache data cluster, filtering the indicators in the first query statement to obtain a filtered first query statement;

[0032] The filtered first query statement is subjected to fragment extraction based on grammatical rules to obtain a first indicator fragment.

[0033] Optionally, the judgment unit is further configured to:

[0034] If there is cached data matching the first query statement, then for the cached data matching the first query statement, determining whether the query time period of the cached data completely includes the query time period of the first query statement; the obtaining supplement unit is further configured to:

[0035] If the query time period of the cached data does not completely include the query time period of the first query statement, original data corresponding to the first query statement is obtained, and the query time period of the missing cached data is supplemented in the cached data cluster.

[0036] Optionally, the acquisition supplement unit is further configured to:

[0037] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, then obtaining the cache interface data corresponding to the first query statement; or

[0038] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rule, and the cache indicator fragment reorganization data is obtained.

[0039] Optionally, the acquisition supplement unit is further configured to:

[0040] If there is no cached data matching the first indicator fragment, original data corresponding to the first query statement is obtained, and the first indicator fragment is supplemented in the cached data cluster.

[0041] Optionally, the acquisition supplement unit is further configured to:

[0042] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, converting the cached data into cache indicator fragment reconstructed data according to the calculation rule, and obtaining the cache indicator fragment reconstructed data; or

[0043] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rules and calculation rules, and the cache indicator fragment reorganization data is obtained.

[0044] In a third aspect, the present application further provides an electronic device, comprising:

[0045] a memory for storing program instructions;

[0046] The processor is configured to call the program instructions stored in the memory and execute the steps of the method according to the obtained program instructions.

[0047] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods described in the first aspect.

[0048] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute any one of the methods described in the first aspect.

[0049] The embodiments of the present application provide a data cache supplement method, apparatus, device and medium, which includes parsing a user's first query statement to obtain a first indicator fragment; judging whether there is cached data matching the first query statement based on the cached data in the cached data cluster; if there is no cached data matching the first query statement, judging whether there is cached data matching the first indicator fragment; if there is cached data matching the first indicator fragment, judging whether the query time period of the cached data completely includes the query time period of the first query statement for the cached data matching the first indicator fragment; if the query time period of the cached data does not completely include the query time period of the first query statement, obtaining the original data corresponding to the first query statement and supplementing the query time period of the missing cached data in the cached data cluster. Since the query time period of the missing cached data can be supplemented in the cached data cluster, the user's new query statement can be more likely to match the cached data, which can reduce the query pressure on the original data, thereby improving the speed of data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 A flow chart of a data cache replenishment method provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a process for filtering indicators provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of two cache data types provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of selecting a reading time range according to an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the complete steps of a data cache replenishment method provided by an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of query statement cache content provided by an embodiment of the present invention;

[0057] Figure 7a A schematic diagram of an indicator fragment cache content provided by an embodiment of the present invention;

[0058] Figure 7b A schematic diagram of another indicator fragment cache content is provided for an embodiment of the present invention;

[0059] Figure 8 A structural diagram of a Prometheus-based cache device provided in an embodiment of the present application;

[0060] Figure 9 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Unless there is a conflict, the embodiments in the present application and the features in the embodiments can be combined with each other in any way. In addition, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that here.

[0062] The terms "first" and "second" in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of its variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in this application can mean at least two, for example, two, three or more, and the embodiments of this application are not limited thereto.

[0063] In the technical solution of this application, the collection, dissemination, and use of data comply with the requirements of relevant national laws and regulations.

[0064] Before introducing a data cache supplement method provided by an embodiment of the present application, for ease of understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0065] As cloud computing clusters grow in size, the deployment of cloud-native components based on Kubernetes has flourished. Prometheus has become widely used for resource monitoring in cloud-native environments. With the widespread deployment of containers, monitoring data is growing exponentially. This presents significant challenges for Prometheus tools, including pre-processing and labeling raw data and storing it. The increasing volume of monitored data often leads to query timeouts and slow queries causing Prometheus service downtime. Optimizing query operations is crucial for greater service stability.

[0066] Prometheus's recording rules allow us to pre-calculate frequently used or computationally intensive expressions and save the results as a new set of data. Querying pre-calculated results is typically much faster than executing the original expression each time it's needed, making it very effective for structured, similar queries. When faced with a large number of similar, complex, and structured queries, aggregating the raw data can hinder query efficiency. Prometheus provides a method for establishing recording rules to simplify complex queries, but currently only offers pre-defined rules. This approach, which imposes a fixed, inflexible rule, also reduces query efficiency.

[0067] In order to improve the speed of data query, the embodiment of the present invention provides a data cache supplement method, such as Figure 1 As shown, the method includes:

[0068] S101: Parse the user's first query statement to obtain a first indicator segment;

[0069] S102: Based on the cached data in the cached data cluster, determine whether there is cached data that matches the first query data;

[0070] S103: If there is no cached data matching the first query statement, determine whether there is cached data matching the first indicator fragment;

[0071] S104: If there is cached data that matches the first indicator fragment, then for the cached data that matches the first indicator fragment, determine whether the query time period of the cached data completely includes the query time period of the first query statement;

[0072] S105. If the query time period of the cached data does not completely include the query time period of the first query statement, obtain the original data corresponding to the first query statement, and supplement the query time period of the missing cached data in the cached data cluster; wherein, the query time period of the missing cached data is the query time period in which the query time period of the first query statement is not within the query time period of the cached data.

[0073] An embodiment of the present application provides a data cache supplement method, which includes parsing a user's first query statement to obtain a first indicator fragment; based on the cached data in the cache data cluster, determining whether there is cached data that matches the first query statement; if there is no cached data that matches the first query statement, determining whether there is cached data that matches the first indicator fragment; if there is cached data that matches the first indicator fragment, then for the cached data that matches the first indicator fragment, determining whether the query time period of the cached data completely includes the query time period of the first query statement; if the query time period of the cached data does not completely include the query time period of the first query statement, obtaining the original data corresponding to the first query statement, and supplementing the query time period of the missing cached data in the cache data cluster. Since the query time period of the missing cached data can be supplemented in the cache data cluster, the user's new query statement can be more likely to match the cached data, which can reduce the query pressure on the original data, thereby improving the speed of data query.

[0074] The data cache supplement method provided by the embodiment of the present application can dynamically generate record rules by identifying the indicator fragments in the user's query statement, thereby guiding the query process to hit the cached data, reducing the query pressure on the original data to improve the speed of data query. Among them, hitting the cached data refers to matching with the cached data, specifically including two situations: the user's first query statement matches the cached data and the first indicator fragment matches the cached data. The first query statement matches the cached data means that there is a second query statement that is identical to the first query statement in the cached data, and the first indicator fragment matches the cached data means that there is a second indicator fragment that is identical to the first indicator fragment in the cached data. When there are multiple first indicator fragments, only all of the multiple first indicator fragments exist in the cached data to match. If only some of the multiple indicator fragments exist, then it cannot match, that is, the indicator fragment fails to hit.

[0075] This application mainly consists of five modules, including the web service module, cache hit judgment module, PromQL time range judgment module, cache Prometheus module and basic Prometheus module. The functions of each module are as follows:

[0076] Web service module: Receives PromQL query statements with time ranges, that is, the user's first query statement. The specific parameter format is similar to the Prometheus external API / api / v1 / query_range.

[0077] Cache hit judgment module: judges whether the first indicator fragment obtained by parsing the first query statement hits the cache, and replaces it with the cache indicator name.

[0078] PromQL time range judgment module: formulates relevant data processing logic based on the unique time series attributes of the data, including splicing, interception, etc.

[0079] Cache Prometheus module: corresponds to the cache indicator name in the cache database and provides statement queries after the original PromQL statement is converted into cache indicators.

[0080] Basic Prometheus module: provides raw data query.

[0081] To facilitate understanding, the following solution is explained using a simple mathematical expression. When users use Prometheus to query data using query statements, Prometheus caches the corresponding query statements and metric fragments. For example, the cached data cluster currently includes the following statement: a+b 8:00-12:00, and the following metric fragments: a8:00-12:00 and b 8:00-12:00. Where a+b represents a query statement, a and b are metric fragments, and 8:00-12:00 represents the query time period.

[0082] When the user uses Prometheus to query data again, the first query statement used is a*b 7:00-9:00. At this time, a*b is different from a+b, which means that the first query statement did not hit the cached data (the first query statement does not match the cached data), but both indicator fragments a and b exist in the cached data, so the first indicator fragment hits the cached data. At this time, compare the query time period. The query time period in the cached data is 8:00-12:00, while the query time period of the first query statement is 7:00-9:00. The two are partially overlapping, that is, the query time period of the cached data does not completely include the query time period of the first query statement. At this time, obtain the original data corresponding to the first query statement, query and return it from the original data, and add the time period 7:00-8:00 where the cached data is missing to the cached data cluster. Then the content included in the cached data cluster can be expanded as follows: a+b 8:00-12:00, and include the following indicator fragments: a 7:00-12:00 and b 7:00-12:00. Next time the user's first query statement is a+b 7:00-9:00, the indicator fragments a 7:00-12:00 and b 7:00-12:00 in the cached data cluster can be re-aggregated to obtain a 7:00-9:00 and b 7:00-12:00 plus a + b 7:00-9:00. The query is performed from the cached data instead of the original data, which can reduce the query pressure on the original data and thus improve the data query speed.

[0083] In a specific implementation, the user's first query statement is parsed to obtain a first indicator fragment, including: filtering the indicators in the first query statement based on the user's first query statement and the existing recording rules in the cache data cluster to obtain a filtered first query statement; and extracting fragments from the filtered first query statement based on grammatical rules to obtain a first indicator fragment.

[0084] For example, the user's first query statement is as follows:

[0085]

[0086] Generally, a PromQL statement has a single metric, but multiple metrics may appear in complex queries. For example, in the example above, there are metric1 and metric2, and operations such as +on() and group_right() are performed.

[0087] like Figure 2 As shown, the original indicator name and the existing record rules in the cache are used to filter the corresponding indicator. Then, the relevant aggregation granularity and indicator name are extracted according to the grammatical rules to form a PromQL grammar fragment. The grammatical rules are mainly:

[0088] agg_fun by(label)metric1{label="xxx"}[tm]

[0089] The above grammatical rules describe that for metric1, first, label filtering is performed to obtain label = "xxx", then the data is extracted at the granularity of t minutes, grouped by label, and finally aggregated using the agg_fun function.

[0090] From this, we can get the relevant attributes of the query statement, as shown in Table 1:

[0091] Table 1

[0092] Indicator name metric1 Label label="xxx" Time dimension [t] Aggregate functions agg_fun Aggregation Tags label

[0093] In this application, the focus is on the indicator metric1 and the time dimension t.

[0094] In summary, PromQL is extracted according to certain grammatical rules to obtain basic indicator fragments. The data extracted from Table 1 is shown in Table 2. The indicator fragments are then synchronized to the cache Prometheus using the recording rules. When the same indicator fragment is obtained after the next query statement is parsed, it can be preliminarily determined that the cached data has been hit.

[0095] Table 2

[0096]

[0097]

[0098] In a specific implementation, after determining whether there is cached data matching the first query statement based on the cached data in the cached data cluster, the method further includes:

[0099] If there is cached data that matches the first query statement, then for the cached data that matches the first query statement, determine whether the query time period of the cached data completely includes the query time period of the first query statement; "completely includes" refers to two situations: "greater than" and "equal to". For example, if the query time period of the cached data is 8:00-10:00 and the query time period of the first query statement is 8:00-9:00, this is "greater than"; if the query time period of the cached data is 8:00-10:00 and the query time period of the first query statement is 8:00-10:00, this is "equal to". Both situations are "completely included".

[0100] If the query time period of the cached data does not completely include the query time period of the first query statement, original data corresponding to the first query statement is obtained, and the query time period of the missing cached data is supplemented in the cached data cluster.

[0101] For example, the cached data includes: a+b 8:00-10:00, and the first query statement is a+b 7:00-9:00. At this time, the first query statement hits the cached data, but the query time periods partially overlap. At this time, the indicator fragments of a 7:00-8:00 and b 7:00-8:00 are added to the cache data cluster to facilitate the next cache hit; for another example, the cached data includes: a+b 8:00-10:00, and the first query statement is a+b 6:00-7:00. At this time, the first query statement hits the cached data, but the query time periods are different. At this time, the indicator fragments of a 6:00-7:00 and b 6:00-7:00 are added to the cache data cluster to facilitate the next cache hit.

[0102] In a specific implementation, after determining whether the query time period of the cached data completely includes the query time period of the first query statement, the method further includes:

[0103] If the query period of the cached data completely includes the query period of the first query statement, and the query period of the cached data is equal to the query period of the first query statement, then the cache interface data corresponding to the first query statement is obtained. In this case, it is equivalent to the first query statement completely hitting the cached data, and the cache interface data can be directly queried and returned; or

[0104] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reconstructed data according to the time rule, and the cache indicator fragment reconstructed data is obtained. For example, the cached data includes: a+b 8:00-10:00, a 8:00-10:00, b 8:00-10:00, and the first query statement is a+b 8:00-9:00. At this time, the first query statement hits the cached data, and a 8:00-9:00 and b 8:00-9:00 are added to obtain a+b 8:00-9:00, thereby obtaining the cache indicator fragment reconstructed data, and the cache indicator fragment reconstructed data can be queried and returned.

[0105] In a specific implementation, after determining whether there is cached data that matches the first indicator fragment, it also includes: if there is no cached data that matches the first indicator fragment, obtaining the original data corresponding to the first query statement, and supplementing the first indicator fragment in the cache data cluster. At this time, it is equivalent to that both the indicator fragment and the query statement fail to hit. For example, the cached data includes: a+b 8:00-10:00, a 8:00-10:00, b 8:00-10:00, and the first query statement is c+d 7:00-9:00. The first indicator fragments obtained through parsing are c 7:00-9:00 and d7:00-9:00. At this time, both the first query statement and the first indicator fragment fail to hit the cache data. The first indicator fragments c 7:00-9:00 and d 7:00-9:00 are directly supplemented to the cache data cluster to facilitate the next cache hit.

[0106] In a specific implementation, after determining whether the query time period of the cached data completely includes the query time period of the first query statement, the method further includes:

[0107] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganized data according to the calculation rule, and the cache indicator fragment reorganized data is obtained. For example, the cached data includes: a+b 8:00-10:00, a 8:00-10:00, b 8:00-10:00, the first query statement is a*b 8:00-10:00, and the first indicator fragments obtained by parsing are a8:00-10:00 and b 8:00-10:00. At this time, the first query statement does not hit the cached data, but the first indicator fragment hits the cached data. At this time, a 8:00-10:00 and b 8:00-10:00 are multiplied to obtain cache indicator fragment reorganized data a*b 8:00-10:00, and the cache indicator fragment reorganized data is obtained, queried and returned; or

[0108] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganized data according to the time rule and calculation rule, and the cache indicator fragment reorganized data is obtained. For example, the cached data includes: a+b 8:00-10:00, a 8:00-10:00, b 8:00-10:00, the first query statement is a*b 8:00-9:00, and the parsed first indicator fragments are a 8:00-9:00 and b 8:00-9:00. At this time, the first query statement does not hit the cached data, but the first indicator fragment hits the cached data. At this time, a 8:00-9:00 and b 8:00-9:00 in the cached data are multiplied to obtain cache indicator fragment reorganized data a*b 8:00-9:00, and the cache indicator fragment reorganized data is obtained, queried and returned.

[0109] The following is a summary of these situations. Figure 3 As shown, the data format in the cache data cluster is primarily key-value, where the key is a query statement with a time range (second query statement) or an extracted metric fragment (second metric fragment). For query statements, the value is data returned by the interface. For extracted metric fragments, the value is the name of the metric recorded in the cache Prometheus, i.e., the reconstructed data of the cache metric fragment. The cache matching rule is to compare the key and the time range, which may result in the following situations as shown in Table 3:

[0110] Table 3

[0111]

[0112] Case A: The key is a query statement, and the query period of the cached data is greater than the query period of the first query statement.

[0113] At this time, the cached data is converted into cached indicator fragment reorganization data according to the time rule, and the cached indicator fragment reorganization data is obtained.

[0114] Case B: The key is an indicator fragment, and the query period of the cached data is greater than the query period of the first query statement.

[0115] At this time, the cached data is converted into cached indicator fragment reorganization data according to the time rule and the calculation rule, and the cached indicator fragment reorganization data is obtained.

[0116] Case C: The key is a query statement, and the query period of the cached data is equal to the query period of the first query statement.

[0117] At this time, the cache interface data corresponding to the first query statement is directly obtained.

[0118] Case D: The key is an index segment, and the query period of the cached data is equal to the query period of the first query statement.

[0119] At this time, the cache data is converted into cache indicator fragment reorganization data according to the calculation rules, and the cache indicator fragment reorganization data is obtained.

[0120] Case E: The key is a query statement, and the query period of the cached data does not completely include the query period of the first query statement.

[0121] At this time, the original data corresponding to the first query statement is obtained, and the query time period in which the cached data is missing is supplemented in the cached data cluster;

[0122] like Figure 4 As shown in the figure, for example, if the cache time range is [9:00–11:00] and the query time range is [8:00–12:00], it is necessary to read data data2 from [8:00-9:00] and [11:00-12:00] and merge it with the cached Prometheus read data data1.

[0123] Case F: The key is an indicator fragment, and the query period of the cached data does not completely include the query period of the first query statement.

[0124] At this time, the original data corresponding to the first query statement is obtained, and the query time period in which the cached data is missing is supplemented in the cached data cluster.

[0125] The above six situations (AF) are all situations where cached data is hit. If the cached data is not hit, that is, there is neither cached data that matches the first query statement nor cached data that matches the first indicator fragment in the cache data cluster, then the original data corresponding to the first query statement is obtained, and the first indicator fragment is supplemented in the cache data cluster.

[0126] In practice, for the various scenarios described above, it's necessary to ensure cached data is updated and data operations in Prometheus are cached. Regarding cached data updates: Because time range records have been added to the cache database, the number of times data is used in different time ranges can be used as a basis for updating the LRU algorithm. Regarding Prometheus data operations: Use tools such as Promtool to write data back and establish relevant recording rules.

[0127] The complete steps of a data cache supplement method provided in an embodiment of the present application are as follows: Figure 5 shown.

[0128] S501: Parse the user's first query statement and obtain a first indicator segment;

[0129] S502: Determine whether the first query statement matches the cached data; if so, execute S508; if not, execute S503;

[0130] S503, determine whether the first indicator fragment matches the cached data; if so, execute S504, if not, execute S507;

[0131] S504: Determine whether the query time period of the cached data matching the first indicator fragment completely includes the query time period of the first query statement; if so, execute S506; if not, execute S505;

[0132] S505: Obtain original data corresponding to the first query statement, and add the missing time period of the cached data matching the first indicator segment to the cached data cluster;

[0133] S506. If the query time period of the cached data is equal to the query time period of the first query statement, then obtain cache indicator fragment reorganized data converted according to the time rule; if the query time period of the cached data is greater than the query time period of the first query statement, then obtain cache indicator fragment reorganized data converted according to the time rule and calculation rule;

[0134] S507: Obtain original data and add the first indicator fragment to the cache data cluster;

[0135] S508: Determine whether the query time period of the cached data matching the first query statement completely includes the query time period of the first query statement; if so, execute S509; if not, execute S510;

[0136] S509: If the query time period of the cached data is equal to the query time period of the first query statement, obtain cache interface data corresponding to the first query statement; if the query time period of the cached data is greater than the query time period of the first query statement, obtain cache indicator fragment reorganization data converted according to the time rule;

[0137] S510: Obtain original data, and add the missing time period of the cached data matching the first query statement to the cached data cluster.

[0138] In this application, the idea of dynamically recording rules and mapping data to a time range is proposed to solve the problem of slow query speed when a large number of frequent similar query statements are performed. The main implementation principle of the cache is to parse the query statement, dynamically establish recording rules in the cache data according to the indicator fragment, and record the correspondence between the indicator fragment and the cache recording rule in the cache database. When a new query request arrives, it will first determine whether it is a hit. If so, it will be calculated and returned from the cache data. Otherwise, it will be queried from the original data and the data that did not hit the cache this time will be synchronized. By adding the cache in the middle, the query speed is improved.

[0139] Furthermore, by introducing the concept of dynamically establishing recording rules, this application can significantly improve Prometheus's ability to provide external query services for a large number of similar and complex queries, while only requiring a small increase in storage space and computing power, thereby ensuring component stability. This can also provide a solution for similar time-series database caching. In addition to using time dimensions and indicator data as cache index keys, cache keys can be appropriately customized based on dimensions such as tags and aggregation functions, depending on different usage scenarios.

[0140] The following is a detailed description of the process of this application using specific examples:

[0141] For example, there are the following servers:

[0142] Prometheus server is on a certain platform, and Prometheus query servers PQ1, PQ2, etc.

[0143] Cache Prometheus server P1;

[0144] Web application server W1;

[0145] Cache hit algorithm server Cache1.

[0146] 1. The web server W1 receives the external query statement parameters and executes the syntax parser to obtain the index fragment data.

[0147] For example, the original query statement is:

[0148] round(sum by(lable1,label2)(increase(metric1{label1="machine 1",label2="address 1",label3="online"}[1m]))or vector(0)+on()group_right()increase(metric2{label1="machine 2",lable2="address 2",label3="offline"}[1m])),1.0).

[0149] The two index fragments extracted are:

[0150] Fragment 1: increase(metric1{label1="machine1",label2="address1",label3="online"}[1m];

[0151] Fragment 2: increase(metric2{label1="machine 2", label2="address 2", label3="offline"}[1m]).

[0152] Then the converted PromQL is:

[0153] round(sum by(lable1,label2)rule1 or vector(0)+on()group_right()rule2),1.0);

[0154] Rule 1 represents segment 1, and rule 2 represents segment 2.

[0155] 2. Cache hit algorithm The server reads the cache data to determine whether the query hits the cache. The contents of the cache database at this time are as follows:

[0156] The query statement cache content is as follows Figure 6 As shown, the indicator fragment buffer content is as follows Figure 7a and Figure 7b shown.

[0157] When performing the following queries, the processing process is as follows:

[0158] (1) The query time is 2022.02.01 9:00-10:00, query statement Q1:

[0159] round(sum by(lable1,label2)(increase(metric1{label1="machine 1",label2="address 1",label3="online"}[1m]))or vector(0)+on()group_right()increase(metric2{label1="machine 2",lable2="address 2",label3="offline"}[1m])),1.0)

[0160] If the query request for 2022.02.01 8:00-10:00 is hit, the converted statement with the recording rule will be executed on cache P1:

[0161] round(sum by(lable1,label2)rule1 or vector(0)+on()group_right()rule2),1.0)

[0162] At this point, just get the data and return it directly.

[0163] (2) The query time is 2022.02.01 9:00-10:00, query statement Q2:

[0164] round(sum by(lable1,label2)(increase(metric1{label1="machine 1",label2="address 1",label3="online"}[1m])or vector(0)+on()group_right()increase(metric3{label1="machine 1",lable2="address 1",label3="offline"}[1m])),1.0);

[0165] The resulting fragment is: increase(metric1{label1="machine 1",label2="address 1",label3="online"}[1m])

[0166] increase(metric3{label1="machine 1", label2="address 1", label3="offline"}[1m])

[0167] If metric 1 matches the cached data but metric 3 fails, the query statement is queried from PQ1 and PQ2 and the results are returned. The data is then synchronized from query servers PQ1 and PQ2 to cache server P1. The cache database also adds the metric fragment cache for metric 3 and the cached data for the entire query statement Q2.

[0168] (3) The query time is 2022.02.01 8:00-10:00, query statement Q1:

[0169] round(sumby(lable1,label2)(increase(metric1{label1="machine 1",label2="address 1",label3="online"}[1m]))or vector(0)+on()group_right()increase(metric2{label1="machine 2",lable2="address 2",label3="offline"}[1m])),1.0)

[0170] Completely hit 2022.02.01 8:00-10:00 and obtain the interface data corresponding to the query statement in the cache database.

[0171] (4) The query time is 2022.02.01 8:00-9:00, query statement Q1:

[0172] round(sumby(lable1,label2)(increase(metric1{label1="machine 1",label2="address 1",label3="online"}[1m]))or vector(0)+on()group_right()increase(metric2{label1="machine 2",lable2="address 2",label3="offline"}[1m])),1.0)

[0173] The time range of the hit indicator fragments Metric1_1m_1649056699 and Metric2_1m_1649056699 is 2022.02.01 8:00-10:00. Query and return them in the cache server P1.

[0174] (5) The query time is 2022.02.01 8:00-11:00, query statement Q1:

[0175] round(sum by(lable1,label2)(increase(metric1{label1="machine 1",label2="address 1",label3="online"}[1m]))or vector(0)+on()group_right()increase(metric2{label1="machine 2",lable2="address 2",label3="offline"}[1m])),1.0)

[0176] If a query statement is matched but the time range exceeds the cache range, the original data PQ1 and PQ2 are queried for the Q1 statement, and the data for the time range of 10:00-11:00 is concatenated with the interface data in the cache database and returned. The Q1 time range is added to the cache database, and the time range data corresponding to the included indicator fragments is synchronized to cache server P1.

[0177] Based on the same inventive concept, the present application embodiment also provides a Prometheus cache device, please refer to Figure 8 , the device comprises:

[0178] The parsing unit 801 is configured to parse a first query statement of a user and obtain a first indicator segment;

[0179] A determination unit 802 is configured to determine, based on cached data in the cached data cluster, whether there is cached data that matches the first query statement; if no cached data matches the first query statement, determine whether there is cached data that matches the first indicator fragment; if there is cached data that matches the first indicator fragment, determine, for the cached data that matches the first indicator fragment, whether a query time period of the cached data completely includes a query time period of the first query statement;

[0180] An acquisition and supplementation unit 803 is used to obtain the original data corresponding to the first query statement if the query time period of the cached data does not completely include the query time period of the first query statement, and supplement the query time period of the missing cached data in the cached data cluster; wherein, the query time period of the missing cached data is a query time period in which the query time period of the first query statement is not within the query time period of the cached data.

[0181] Optionally, the parsing unit 801 is specifically configured to:

[0182] Based on the user's first query statement and the existing recording rules in the cache data cluster, filtering the indicators in the first query statement to obtain a filtered first query statement;

[0183] The filtered first query statement is subjected to fragment extraction based on grammatical rules to obtain a first indicator fragment.

[0184] Optionally, the judging unit 802 is further configured to:

[0185] If there is cached data matching the first query statement, then for the cached data matching the first query statement, determining whether the query time period of the cached data completely includes the query time period of the first query statement; the obtaining supplement unit is further configured to:

[0186] If the query time period of the cached data does not completely include the query time period of the first query statement, original data corresponding to the first query statement is obtained, and the query time period of the missing cached data is supplemented in the cached data cluster.

[0187] Optionally, the acquisition supplement unit 803 is further configured to:

[0188] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, then obtaining the cache interface data corresponding to the first query statement; or

[0189] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rule, and the cache indicator fragment reorganization data is obtained.

[0190] Optionally, the acquisition supplement unit 803 is further configured to:

[0191] If there is no cached data matching the first indicator fragment, original data corresponding to the first query statement is obtained, and the first indicator fragment is supplemented in the cached data cluster.

[0192] Optionally, the acquisition supplement unit 803 is further configured to:

[0193] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, converting the cached data into cache indicator fragment reconstructed data according to the calculation rule, and obtaining the cache indicator fragment reconstructed data; or

[0194] If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rules and calculation rules, and the cache indicator fragment reorganization data is obtained.

[0195] Based on the same inventive concept, the embodiment of the present application provides an electronic device that can implement the function of the Prometheus data cache supplement method discussed above. Please refer to Figure 9 , the device includes a processor 901 and a memory 902.

[0196] Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium, a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to execute any of the data cache replenishment methods discussed above. Because the principles underlying the problems solved by the computer-readable storage medium are similar to those of the data cache replenishment method, the implementation of the computer-readable storage medium can be referenced to the implementation of the method, and any repetitions will not be repeated here.

[0197] Based on the same inventive concept, embodiments of the present application further provide a computer program product, comprising computer program code that, when executed on a computer, causes the computer to perform any of the data cache replenishment methods discussed above. Because the principles underlying the problems solved by the aforementioned computer program products are similar to those of the data cache replenishment methods, the implementation of the aforementioned computer program products can be referenced to the implementation of the methods, and any repetitions will not be repeated.

[0198] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0199] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0200] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0202] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data cache replenishment method, characterized in that: include: Parsing the user's first query statement to obtain a first indicator fragment; Based on the cached data in the cached data cluster, determining whether there is cached data that matches the first query statement; If there is no cached data matching the first query statement, determining whether there is cached data matching the first indicator fragment; If there is cached data that matches the first indicator fragment, determining, for the cached data that matches the first indicator fragment, whether the query time period of the cached data completely includes the query time period of the first query statement; If the query time period of the cached data does not completely include the query time period of the first query statement, obtaining the original data corresponding to the first query statement, and supplementing the query time period of the missing cached data in the cached data cluster; The query time period in which the cached data is missing is a query time period in which the query time period of the first query statement is not within the query time period of the cached data.

2. The method according to claim 1, wherein The step of parsing the user's first query statement to obtain a first indicator segment includes: Based on the user's first query statement and the existing recording rules in the cache data cluster, filtering the indicators in the first query statement to obtain a filtered first query statement; The filtered first query statement is subjected to fragment extraction based on grammatical rules to obtain a first indicator fragment.

3. The method according to claim 1, wherein After determining whether there is cached data matching the first query statement based on the cached data in the cached data cluster, the method further includes: If there is cached data matching the first query statement, determining, for the cached data matching the first query statement, whether a query time period of the cached data completely includes a query time period of the first query statement; If the query time period of the cached data does not completely include the query time period of the first query statement, original data corresponding to the first query statement is obtained, and the query time period of the missing cached data is supplemented in the cached data cluster.

4. The method according to claim 3, wherein After determining whether the query time period of the cached data completely includes the query time period of the first query statement, the method further includes: If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, then obtaining the cache interface data corresponding to the first query statement; or If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rule, and the cache indicator fragment reorganization data is obtained.

5. The method according to claim 1, wherein After determining whether there is cached data that matches the first indicator segment, the method further includes: If there is no cached data matching the first indicator fragment, original data corresponding to the first query statement is obtained, and the first indicator fragment is supplemented in the cached data cluster.

6. The method according to claim 1, wherein After determining whether the query time period of the cached data completely includes the query time period of the first query statement, the method further includes: If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is equal to the query time period of the first query statement, converting the cached data into cache indicator fragment reconstructed data according to the calculation rule, and obtaining the cache indicator fragment reconstructed data; or If the query time period of the cached data completely includes the query time period of the first query statement, and the query time period of the cached data is greater than the query time period of the first query statement, the cached data is converted into cache indicator fragment reorganization data according to the time rules and calculation rules, and the cache indicator fragment reorganization data is obtained.

7. A data cache supplement device, characterized in that: include: A parsing unit, configured to parse a first query statement of a user and obtain a first indicator segment; a determining unit, configured to determine whether there is cached data matching the first query statement based on cached data in the cached data cluster; If there is no cached data matching the first query statement, determining whether there is cached data matching the first indicator fragment; if there is cached data matching the first indicator fragment, determining, for the cached data matching the first indicator fragment, whether the query time period of the cached data completely includes the query time period of the first query statement; An acquisition and supplementation unit is used to obtain the original data corresponding to the first query statement if the query time period of the cached data does not completely include the query time period of the first query statement, and supplement the query time period of the missing cached data in the cached data cluster; wherein, the query time period of the missing cached data is a query time period in which the query time period of the first query statement is not within the query time period of the cached data.

8. An electronic device, characterized in that: include: a memory for storing program instructions; A processor is configured to call the program instructions stored in the memory, and execute the steps included in the method according to any one of claims 1 to 6 according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images