Data query method and device, storage medium and program product

Abstract

The embodiment of the invention provides a data query method and device, a storage medium and a program product. In the query processing method, aiming at a target sub-query statement in an original query statement, a plurality of storage nodes can be utilized to execute data reading operation in parallel according to a query condition in the target sub-query statement, and local calculation corresponding to a target aggregation operator in the target sub-query statement is executed on the read data, so that the target sub-query statement in the original query statement is obtained. And obtaining a local calculation result. Furthermore, the local calculation result of the read data instead of the read data is returned to the calculation node by the storage node, so that the data volume of data transmission between the storage node and the calculation node can be reduced, and the consumption of network bandwidth and the calculation time consumption caused by a large amount of data transmission are reduced; in addition, the storage node is used for sharing part of the calculation amount, and the query efficiency can be further improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data query method, device, storage medium, and program product. Background Technology

[0002] In time-series data processing scenarios, there exists an open-source monitoring project that supports the collection, storage, and computation of indicator data, and also provides data visualization capabilities. Users can query this indicator data using query statements that conform to the specified syntax of the monitoring project, and trigger alarms based on this data. However, in some monitoring systems built upon this project, the computation engine only supports single-machine, single-coroutine execution of queries, resulting in low query efficiency and failing to meet high real-time requirements in scenarios with large amounts of data. Therefore, a new solution is needed. Summary of the Invention

[0003] This application provides a data query method, device, storage medium, and program product for monitoring the query efficiency of a system.

[0004] This application provides a data query method applied to a computing node. The data is stored in a storage layer, which includes multiple storage nodes. The method includes: receiving a computation request corresponding to an original query statement; identifying a target subquery statement in the original query statement, the target subquery statement consisting of query conditions and a target aggregation operator nested outside the query conditions; rewriting the target subquery statement and generating an execution plan based on the rewritten target subquery statement to push down the computation corresponding to the query conditions to the storage layer; sending the execution plan to multiple storage nodes in the storage layer, the execution plan enabling the multiple storage nodes to perform data reading operations in parallel according to the query conditions, and to perform local computations corresponding to the target aggregation operator on the read data in parallel to obtain local computation results; and merging the local computation results returned by the multiple storage nodes according to the target aggregation operator to obtain the computation result of the target subquery statement.

[0005] Optionally, identifying the target subquery statement in the original query statement includes: obtaining the syntax tree corresponding to the original query statement, wherein the nodes in the syntax tree correspond to the aggregation operators in the original query statement, and the leaf nodes correspond to the query conditions in the original query statement; traversing the nodes in the syntax tree, and identifying target nodes that satisfy set conditions during the traversal process, wherein the set conditions include: the child nodes of the target node are leaf nodes; and determining the target subquery statement based on the operators corresponding to the target node and the query conditions corresponding to the child nodes of the target node.

[0006] Optionally, the target subquery statement is rewritten, and an execution plan is generated based on the rewritten target subquery statement to push down the calculation corresponding to the query condition to the storage layer. This includes: the target aggregation operator and a built-in subquery statement embedded in the target aggregation operator. The built-in subquery statement includes the query condition and a local aggregation operator nested outside the query condition. The local aggregation operator is used to perform the local calculation corresponding to the target aggregation operator. During the calculation of the rewritten target subquery statement, the execution plan is generated based on the built-in subquery statement.

[0007] Optionally, rewriting the target subquery statement includes: rewriting the target subquery statement into a built-in subquery statement corresponding to the target aggregation operator.

[0008] Optionally, after rewriting the target subquery statement into the built-in subquery statement corresponding to the target aggregation operator, the method further includes: adding rewriting identifier information corresponding to the built-in subquery statement to the rewritten target subquery statement; generating an execution plan based on the built-in subquery statement during the calculation of the rewritten target subquery statement, including: parsing the rewritten target subquery statement during the calculation of the rewritten target subquery statement; if the rewriting identifier information is parsed into the rewritten target subquery statement, generating the execution plan based on the built-in subquery statement corresponding to the rewriting identifier information.

[0009] Optionally, it further includes: responding to an expansion event, obtaining a newly added storage node, the newly added storage node being used to receive and store newly written data together with the plurality of storage nodes; sending the execution plan to the newly added storage node, the execution plan being used to enable the newly added storage node to perform a data reading operation according to the query conditions, and to perform local calculations on the read data according to the local aggregation operator.

[0010] This application also provides a data query method applied to a storage node, comprising: receiving an execution plan sent by a computing node, the execution plan being generated based on a built-in subquery statement of a target aggregation operator, the built-in subquery statement including: query conditions and a local aggregation operator nested outside the query conditions, the local aggregation operator being used to perform local calculations corresponding to the target aggregation operator; performing a data reading operation according to the data query conditions in the execution plan; performing local calculations on the read data according to the local aggregation operator to obtain a local calculation result; and returning the local calculation result to the computing node.

[0011] Optionally, the storage node runs a computing engine; according to the target aggregation operator, performing local calculations on the read data to obtain local calculation results includes: calling the computing engine to perform local calculations on the read data according to the local aggregation operator to obtain local calculation results.

[0012] This application also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to perform the steps in the method provided in this application.

[0013] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the method provided in this application.

[0014] This application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in this application.

[0015] In the data query method provided in this application embodiment, for the target subquery statement in the original query statement, multiple storage nodes can be used to perform data reading operations in parallel according to the query conditions in the target subquery statement, and perform local calculations on the read data corresponding to the target aggregation operator in the target subquery statement to obtain the local calculation results. Furthermore, the storage node returns the read data to the computing node not the read data itself, but the local calculation results of the read data. This reduces the amount of data transmitted between the storage node and the computing node, thereby reducing network bandwidth consumption and computation time caused by large data transmissions, and improving query efficiency. In addition, multi-node parallel computing can be achieved based on multiple storage nodes, which greatly improves computational efficiency compared to the existing single-node multi-coroutine execution method, thus adapting to massive computing scenarios. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a schematic diagram of the structure of a monitoring system that incorporates multi-machine parallel computing at the computing layer;

[0018] Figure 2 This is a schematic diagram of the structure of a monitoring system that performs multi-coroutine queries within the main computing node;

[0019] Figure 3 This is a schematic diagram of the structure of a monitoring system provided in an exemplary embodiment of this application;

[0020] Figure 4 This is a flowchart illustrating the execution of the data query method provided in an exemplary embodiment of this application on the computing node side;

[0021] Figure 5 This is a schematic diagram of the syntax tree structure provided in an exemplary embodiment of this application;

[0022] Figure 6 This is a flowchart illustrating the execution of the data query method provided in an exemplary embodiment of this application on either storage node side;

[0023] Figure 7 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0026] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0027] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0028] In some monitoring systems built on open-source monitoring projects, the computing engine only supports single-machine, single-coroutine execution of queries, resulting in low query efficiency and failing to meet high real-time requirements in scenarios with large amounts of data. Take the Prometheus project as an example. Prometheus is an open-source monitoring project for processing time-series data. In this project, metric data consists of four types of information: timestamp, metric name, tag list, and metric value. The "metric name" and "tag list" uniquely identify a timeline. Users can use query statements that conform to the specified syntax of this monitoring project to query this metric data. In the Prometheus project, the main performance overhead when executing query statements is reflected in data transmission, serialization and deserialization, computation execution, and garbage collection (GC) in Go (a statically compiled language).

[0029] To improve query efficiency, some improvements based on the Prometheus project have introduced parallel execution architectures during the query execution phase. For example, some improved solutions use Object Storage Service (OSS) to store metric data and introduce a multi-machine parallel computing architecture at the computation layer. Figure 1 As shown, this architecture may include a main compute node and multiple sub-compute nodes. In this scheme, the execution of a query statement mainly includes the following steps: sub-compute nodes read raw data from OSS data in parallel, perform serialization / deserialization operations, sub-compute nodes perform computations in parallel, perform serialization / deserialization operations, and the main compute node performs summary computations. Although this parallel architecture can significantly improve computational performance, the overhead caused by data transmission, serialization / deserialization, and GC operations is still very large.

[0030] In some improved solutions, the execution engine that follows the native query syntax of the monitoring project is abandoned at the computation layer, and an independent query syntax is designed. This independent query syntax is not fully compatible with the native query syntax of the monitoring project. Therefore, to some extent, the results obtained from queries based on this independent query syntax are incompatible with the monitoring project. Furthermore, as... Figure 2 As shown, this architecture includes a master computing node, and its parallel computation is reflected in multi-coroutine queries within a single node. In scenarios with massive timelines, single-node computation can severely impact the stability of the master node's service processes, meaning that the CPU or memory usage of the service processes may continuously spike.

[0031] To address the aforementioned technical problems, a solution is provided in some embodiments of this application. The technical solutions provided by each embodiment of this application are described in detail below with reference to the accompanying drawings.

[0032] Figure 3 This is a schematic diagram of the structure of a monitoring system provided in an exemplary embodiment of this application, as shown below. Figure 3 As shown, the monitoring system 30 may include a computing node 31 and multiple storage nodes 32 in the storage layer.

[0033] In this context, computing node 31 refers to the basic unit that undertakes data processing and computing tasks in a distributed computing environment or cluster architecture. Such nodes typically possess independent computing resources, such as a central processing unit (CPU), memory, and storage space, and are capable of executing the computing tasks or parts of the tasks assigned to them. This computing node can be a physical server, an elastic computing instance, or a virtual machine; this embodiment does not impose any limitations.

[0034] The monitoring system 30 can be applied in monitoring and early warning scenarios. In these scenarios, the computing node 31 can perform data capture, query processing, and rule evaluation operations. When performing data capture, the computing node 31 can capture indicator data from a configured target object, such as an application, service, or hardware device, triggered by a set event (e.g., a timed event). The target object can expose an address or endpoint to the computing node 31, allowing the computing node 31 to obtain indicator data from the target object based on that address and endpoint.

[0035] In some scenarios, a log monitoring service can be deployed on compute node 31. This service can capture performance metrics of the target object by monitoring its log data. These performance metrics may include, but are not limited to, the target object's CPU utilization, memory utilization, disk I / O (Input / Output), network traffic, request latency, error rate, throughput, and other metrics. The captured metric data is stored on storage node 32 in time-series format.

[0036] When performing query processing operations, compute node 31 can receive query requests, execute the query operations corresponding to the query statement, and return results. The query operations corresponding to the query statement can include various calculations and aggregation operations on the time series data. Specifically, the calculation operations can include: calculating the average rate over a certain time range, calculating the increment (increase()) over a certain time range, selecting the k largest values ​​from a vector, selecting the k smallest values ​​from a vector, and calculating quantiles over a certain time range. The aggregation operations can include: calculating the sum of all selected time series, calculating the average of all selected time series, finding the minimum value (min) among all selected time series, finding the maximum value (max()) among all selected time series, and calculating the number of selected time series.

[0037] When performing rule evaluation operations, compute node 31 can pre-compute and store frequently used query results based on configured record rules, and detect specific conditions and trigger alarms based on alarm rules.

[0038] Storage node 32 in the storage layer can be a virtual node or a physical node. A virtual node refers to a logical storage device created through virtualization technology, which typically runs on a physical host. A physical node refers to an actual physical device, such as a server, storage array, or network attached storage (NAS) device. Storage node 32 is used to store the indicator data captured by computing node 31 in a local file system. This indicator data is organized in the form of time series, with each time series consisting of four types of information: timestamp, indicator name, tag list, and indicator value. The indicator name and tag list are used to uniquely identify a time series, and a single time series is called a timeline. Storage node 32 can read data according to the data read instructions of computing node 31 and return the read data to computing node 31. In this embodiment, storage node 32 is further used to: perform specified calculations on the data after reading it, so as to distribute the data calculation tasks to storage nodes for execution, thereby improving the overall efficiency and stability of monitoring system 30.

[0039] Figure 3 The illustrated monitoring system 30 can perform a query method for indicator data. The following will, with reference to the accompanying drawings, provide an exemplary description of the query method from the perspectives of the computing node 31 and the storage node 32.

[0040] Figure 4 This is a flowchart illustrating the execution of a data query method provided in an exemplary embodiment of this application on the computing node side. The method may include, for example: Figure 4 The steps shown are as follows:

[0041] Step 401: Receive the calculation request corresponding to the original query statement.

[0042] Step 402: In the original query statement, identify the target subquery statement, which consists of query conditions and target aggregation operators nested outside the query conditions.

[0043] Step 403: Rewrite the target subquery statement and generate an execution plan based on the rewritten target subquery statement to push down the calculation corresponding to the query conditions to the storage layer.

[0044] Step 404: Send the execution plan to multiple storage nodes in the storage layer. The execution plan is used to enable the multiple storage nodes to perform data reading operations in parallel according to the query conditions, and to perform local calculations corresponding to the target aggregation operator on the read data in parallel to obtain local calculation results.

[0045] Step 405: Based on the target aggregation operator, merge the local calculation results returned by the multiple storage nodes to obtain the calculation result of the target sub-query statement.

[0046] In step 401, the computation request may be sent by the user through the client of the computing node 31, or it may be sent by other applications or services through the communication interface provided by the computing node 31. This embodiment does not impose any restrictions.

[0047] The expression in the original query statement can contain various elements, such as vector selectors, function calls, and aggregation operators. Vector selectors are typically used to select data points from time-series databases. Vector selectors can include unlabeled vector selectors and labeled vector selectors. For example, `up` is an unlabeled vector selector used to read the `up` status of a monitored target, indicating whether the monitored target is operating normally. Similarly, `http_requests_total{job="api-server"}` is a labeled vector selector used to select time series data for the `http_requests_total` metric labeled `job="api-server"`.

[0048] Function calls are used to perform calculations on vectors selected by a vector selector. For example, the `rate()` function calculates the average rate over a certain time range for the selected vectors; the `increase()` function calculates the increment over a certain time range for the selected vectors; the `topk()` function selects the k largest values ​​from the selected vectors; the `bottomk()` function selects the k smallest values ​​from the selected vectors; and the `quantile_over_time()` function calculates the quantiles over a certain time range for the selected vectors. For ease of description, function calls will be labeled as `F()`.

[0049] Aggregation operators are used to summarize the calculation results of vectors selected by the vector selector or corresponding function calls, and return the result, which is usually a new vector. For example, the aggregation operators sum() calculate the sum of all selected time series, avg() calculates the average of all selected time series, min() finds the minimum value of all selected time series, max() finds the maximum value of all selected time series, and count() counts the number of selected time series.

[0050] For example, the expression corresponding to the original query statement is max(rate(up[1m])), where up[1m] is the time series of the up index of the monitored object in the past minute. rate(up[1m]) is used to calculate the rate of change of the up index of the monitored object in the past minute, and max(...) is used to take the maximum value of the rate of change of the up index of the returned monitored object.

[0051] In step 402, compute node 31 can identify a target subquery statement within the original query statement. This target subquery statement refers to a subquery statement composed of query conditions and target aggregation operators nested outside of those query conditions. "Target aggregation operator nested outside of query conditions" means that the target aggregation operator is directly nested outside the query conditions, and there is no other hierarchical relationship between the target aggregation operator and the query conditions. That is, the target aggregation operator is used to directly calculate the data retrieved based on the query conditions, and typically does not involve cross-storage node calculations. Therefore, the calculations corresponding to the target aggregation operator are suitable to be split into local calculations performed by storage nodes and summary calculations performed by compute node 31.

[0052] In this embodiment, the query condition may include a vector selector, or it may include a vector selector and a function call nested outside the vector selector. For example, suppose a subquery in the original query statement is max(rate(up[1m])), in which the target aggregation operator is the max operator, the query condition is rate(up[1m]), and the max operator is nested outside the query condition. Therefore, this subquery statement can be used as the target subquery statement.

[0053] In some optional embodiments, compute node 31 can identify the target subquery from the original query statement using rule matching or keyword matching. For example, compute node 31 can establish matching rules based on commonly used vector selectors and function calls, and filter out vector selectors or function calls from the first subquery statement based on these matching rules. Then, it can be determined whether the filtered vector selector or function call is nested with an aggregation operator. If the filtered vector selector is nested with an aggregation operator, the subquery formed by the vector selector and its nested aggregation operator can be used as the target subquery. Alternatively, if the filtered function call is nested with an aggregation operator, the subquery formed by the function call and its nested aggregation operator can be used as the target subquery.

[0054] In some alternative embodiments, the compute node 31 can identify the target subquery from the original query statement using a syntax tree traversal. Specifically, the compute node 31 can obtain the syntax tree corresponding to the original query statement, where the nodes in the syntax tree correspond to the aggregate operators in the original query statement, and the leaf nodes correspond to the query conditions in the original query statement. Based on the format of the query statement, it can be determined that when the child node of the target node is a leaf node, the operator corresponding to the target node is usually an aggregate operator nested outside the query conditions. Based on this, the compute node 31 can traverse the nodes in the syntax tree and, during the traversal, identify the target node that satisfies a set condition, which is that the child node of the target node is a leaf node. After determining the target node, the compute node 31 can determine the target subquery statement based on the operator corresponding to the target node and the query conditions corresponding to the child node of the target node.

[0055] The following will provide a concrete example to further illustrate the method of identifying target subqueries based on syntax trees. Assume the original query expression is:

[0056] The expression avg{sum1{sum2{max2(rate(a[1m]))+min(rate(b[1m]))}+max1(rate(c[1m]))}} can be used to draw... Figure 5 The syntax tree shown. (As shown in the image) Figure 5As shown, the vertices in the syntax tree are the outermost aggregate operator avg in the expression, the child nodes of the root node are the aggregate operator sum1, the child nodes of the aggregate operator sum1 are the aggregate operators sum2 and max1, the child nodes of the aggregate operator sum2 are the aggregate operators max2 and min, and the leaf nodes are the query conditions nested inside the aggregate operators max1, max2, and min respectively.

[0057] Node 31 can traverse the nodes in the syntax tree and, during the traversal, identify target nodes that meet set conditions. For example... Figure 5 As shown, in the syntax tree, the child nodes of the aggregation operators max1, max2, and min are all leaf nodes. Therefore, the aggregation operators max1, max2, and min can be used as target aggregation operators, respectively. The target subqueries identified from this query statement can be: max2(rate(a[1m])), min(rate(b[1m])), and max1(rate(c[1m])). The query methods for the target subqueries corresponding to each aggregation operator are the same or similar. In this embodiment, only one target aggregation operator is used as an example to illustrate the query method.

[0058] Based on the above implementation method, by using rule matching or syntax tree analysis, the target subquery statement that can be split into merged calculation and local calculation can be accurately identified from the query statement, thereby facilitating the subsequent implementation of local calculation with the help of storage nodes and reducing the amount of data returned from storage nodes to computing nodes.

[0059] After determining the target subquery, in step 403, computation node 31 can utilize multiple storage nodes to perform data reading operations in parallel according to the query conditions, and perform local calculations corresponding to the target aggregation operator on the read data to obtain the local calculation results. In this implementation, the data reading operation corresponding to the target subquery is pushed down to the multiple storage nodes for execution, and the partial calculation operations corresponding to the target subquery are also pushed down to the multiple storage nodes for execution.

[0060] In some optional embodiments, the computing node 31 can send the query conditions in the target subquery statement to the plurality of storage nodes, so that the plurality of storage nodes can perform data reading operations according to the query conditions. After the plurality of storage nodes read the data, the computing node 31 can send calculation instructions to the plurality of storage nodes according to the target aggregation operator in the target subquery statement, so that the plurality of storage nodes can perform calculation operations on the read data according to the calculation instructions and return the calculation results.

[0061] In some alternative embodiments, computing node 31 may rewrite the target subquery statement and generate an execution plan during the execution of the rewritten subquery statement. This execution plan enables any storage node to perform data reading operations based on the query conditions and to perform computational processing on the read data. Optionally, the rewritten target subquery statement may include a target aggregation operator and a built-in subquery statement embedded within the target aggregation operator. The built-in subquery statement includes the query conditions and a local aggregation operator nested outside the query conditions. The local aggregation operator is used to perform local calculations corresponding to the target aggregation operator. For example, if the target aggregation operator is a sum operator, the built-in subquery statement of the sum operator may include the query conditions and a local aggregation operator of the sum operator nested outside the query conditions. The local aggregation operator of the sum operator is used to perform partial summation operations on the storage node. As another example, if the target aggregation operator is a max operator, the built-in subquery statement of the max operator may include the query conditions and a local aggregation operator of the max operator nested outside the query conditions. The local aggregation operator of the max operator is used to perform partial maximum value operations on the storage node.

[0062] Optionally, when rewriting the target subquery statement, the computing node 31 can rewrite the target subquery statement into the built-in subquery statement corresponding to the target aggregation operator. For example, assuming the target subquery statement is max(F1()), the rewritten target subquery statement is max(max(F1())), that is, max(F1()) is rewritten into the built-in subquery statement of the max operator. Based on this implementation, the computation operation of the target aggregation operator can be split into local computations that can be executed by the storage node and merged computations that can be executed by the computing node 31, thereby improving the parallelism of the computation.

[0063] During the computation of the rewritten target subquery, compute node 31 can generate an execution plan based on the built-in subquery and send this plan to the multiple storage nodes. This execution plan enables the multiple storage nodes to perform data reading operations in parallel according to the query conditions, and to perform local calculations on the read data based on the local aggregation operators in the built-in subquery. Taking the rewritten built-in subquery as max(F1()) as an example, the execution plan generated based on max(F1()) may include: performing calculations on the read data according to the function call F1(), and performing a maximum value operation on the calculated data according to max. For any storage node, the maximum value calculation can be performed locally according to this execution plan, thereby reducing the computational burden on the compute node. Furthermore, the storage node can return the calculated local maximum value to the compute node instead of directly returning the data read according to the query conditions, saving several times the amount of data transmission.

[0064] Optionally, the rewriting operation of the target subquery statement by the compute node 31 can be performed in the preprocessing stage, which is located before the execution stage of the original query statement. In the preprocessing stage, after rewriting the target subquery statement into the built-in subquery statement corresponding to the target aggregation operator, the compute node 31 can add rewriting identifier information corresponding to the built-in subquery statement to the rewritten target subquery statement, so that the built-in subquery statement can be specially processed when it is executed in the execution stage. For example, assuming the target subquery statement is max(F1()), the rewritten target subquery statement is max(PD=max(F1())), where PD is the rewriting identifier information used to indicate that the aggregation calculation corresponding to the built-in subquery statement needs to be pushed down to the storage node for local execution.

[0065] For example, suppose the expression corresponding to a certain query statement is:

[0066] The original query statement, avg{sum1{sum2{max2(F1)+min(F2)}+max1(F3)}}, has the following rewritten expression:

[0067] avg{sum1{sum2{max2(PD=max2(F1))+min(PD=min(F2))}+max1(PD=max1(F3))}}.

[0068] Accordingly, during the computation of the rewritten target subquery statement, computing node 31 can parse the rewritten target subquery statement. If computing node 31 parses rewrite identification information in the rewritten target subquery statement, it generates an execution plan based on the built-in subquery statement corresponding to the rewrite identification information. Taking the above query statement as an example, during the execution of the query statement, the "PD" identification information in the query statement can be identified, and an execution plan for the built-in subquery statement corresponding to the "PD" identification information can be generated. For example, an execution plan corresponding to max2(F1), an execution plan corresponding to min(F2), and an execution plan corresponding to max1(F3) can be generated.

[0069] After generating the execution plan, compute node 31 can send query requests to multiple storage nodes according to the generated execution plan, so that the multiple storage nodes can perform data reading operations and local aggregation calculations corresponding to the query conditions according to the execution plan. Furthermore, for any target aggregation operator, its corresponding aggregation operation is split into parallel local aggregation executed by multiple storage nodes and summary aggregation executed by compute node 31, which greatly improves aggregation efficiency.

[0070] In step 404, the compute node 31 can receive the local calculation results returned by the multiple storage nodes, and merge the local calculation results returned by the multiple storage nodes according to the target aggregation operator to obtain the calculation result of the target subquery statement. For example, after obtaining the local maximum value returned by each of the multiple nodes, the compute node 31 can perform a maximum value operation on the multiple local maximum values ​​to obtain the final calculation result of the maximum value.

[0071] In this implementation, the storage node returns a partial computation result of the read data to the compute node, instead of the read data itself. This reduces the amount of data transferred between the storage and compute nodes, lowers network bandwidth consumption, and reduces computation time caused by large data transfers, thus improving query efficiency. Furthermore, multi-node parallel computing can be achieved based on multiple storage nodes. Compared to the existing single-node multi-coroutine execution method, this significantly improves computational efficiency and makes it suitable for massive computing scenarios.

[0072] It is worth noting that in the monitoring system 30 provided in this application embodiment, the number of storage nodes can be flexibly expanded according to needs. This flexible expansion refers to increasing the number of storage nodes in the monitoring system 30 to improve parallel computing capabilities. An exemplary description will follow.

[0073] Optionally, compute node 31 can respond to an expansion event and acquire a newly added storage node. This new storage node, along with the existing storage nodes, receives and stores newly written data. The newly written data refers to new metric data captured by compute node 31 after the addition of the new storage node. Compute node 31 can send an execution plan to the newly added storage node. This execution plan enables the new storage node to perform data reading operations based on query conditions and to perform local calculations on the read data based on local aggregation operators.

[0074] Optionally, the expansion event may be an event in which the computing node 31 detects that the current load pressure of multiple storage nodes in the monitoring system 30 is greater than a set pressure threshold, or it may be an event in which the computing node 31 detects that the average feedback latency of the multiple storage nodes is greater than a set latency threshold, or it may be a timed event. This embodiment does not impose any restrictions.

[0075] Based on this implementation, when the computational pressure on the multiple storage nodes is high, the computing node 31 can flexibly expand the number of storage nodes, thereby improving the computational performance of the storage layer and thus helping to improve query stability in massive computing scenarios.

[0076] Figure 6This is a flowchart illustrating the execution of a data query method provided in an exemplary embodiment of this application on any storage node. The any storage node can be... Figure 3 Storage node 32 is shown. (As shown) Figure 6 As shown, this method mainly includes the following steps:

[0077] Step 601: Receive the execution plan sent by the computing node. The execution plan is generated based on the built-in subquery statement of the target aggregation operator. The built-in subquery statement includes: query conditions and local aggregation operators nested outside the query conditions. The local aggregation operators are used to perform local calculations corresponding to the target aggregation operator.

[0078] Step 602: Perform a data reading operation according to the data query conditions in the execution plan.

[0079] Step 603: Perform local calculations on the read data according to the local aggregation operator to obtain the local calculation results.

[0080] Step 604: Return the local calculation results to the calculation node.

[0081] In this embodiment, the specific method of generating the execution plan received by the storage node can be referred to the description in the foregoing embodiments, and will not be described again here.

[0082] After obtaining the execution plan, the storage node can perform data reading operations based on the data query conditions in the execution plan, and perform local calculations on the read data according to the local aggregation operators in the execution plan to obtain local calculation results. Optionally, unlike traditional storage nodes, in this embodiment, the storage node runs a computing engine. This computing engine is custom-developed using a specific computer programming language (e.g., C++) based on the execution principle of query statements. After performing the data reading operation, the storage node can call this computing engine to perform local calculations on the read data according to the local aggregation operators to obtain local calculation results. The storage node can return the local calculation results to the computing node 31 for the computing node 31 to perform a merging operation.

[0083] In this implementation, the storage node returns to the compute node not the read data, but a partial calculation result of the read data. This reduces the amount of data transferred between the storage node and the compute node, reduces network bandwidth consumption and computation time caused by large data transfers, and improves query efficiency.

[0084] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 501 to 504 can be device A; or the execution subject of steps 501 and 502 can be device A, and the execution subject of step 503 can be device B; and so on.

[0085] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 501, 502, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0087] Figure 7 This application illustrates a schematic diagram of the structure of an electronic device provided in an exemplary embodiment, as shown below. Figure 7 As shown, the electronic device includes: a memory 701, a processor 702, and a communication component 703.

[0088] Memory 701 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0089] In some exemplary embodiments, Figure 7The illustrated electronic device is used to deploy computing nodes, processor 702, coupled to memory 701, for executing computer programs in memory 701 to: receive a computation request corresponding to an original query statement via communication component 703; identify a target subquery statement in the original query statement, the target subquery statement consisting of query conditions and target aggregation operators nested outside the query conditions; rewrite the target subquery statement and generate an execution plan based on the rewritten target subquery statement for pushing down the computation corresponding to the query conditions to the storage layer; send the execution plan to multiple storage nodes in the storage layer, the execution plan being used to enable the multiple storage nodes to perform data reading operations in parallel according to the query conditions, and to perform local computations corresponding to the target aggregation operators on the read data in parallel, to obtain local computation results; and merge the local computation results returned by the multiple storage nodes according to the target aggregation operators to obtain the computation result of the target subquery statement.

[0090] Optionally, when the processor 702 identifies the target subquery statement in the original query statement, it specifically performs the following steps: obtaining the syntax tree corresponding to the original query statement, wherein the nodes in the syntax tree correspond to the aggregation operators in the original query statement, and the leaf nodes correspond to the query conditions in the original query statement; traversing the nodes in the syntax tree, and identifying target nodes that satisfy set conditions during the traversal process, wherein the set conditions include: the child nodes of the target node are leaf nodes; and determining the target subquery statement based on the operators corresponding to the target node and the query conditions corresponding to the child nodes of the target node.

[0091] Optionally, when the processor 702 rewrites the target subquery statement and generates an execution plan based on the rewritten target subquery statement to push down the calculation corresponding to the query condition to the storage layer, it specifically performs the following: rewriting the target subquery statement, wherein the rewritten target subquery statement includes: the target aggregation operator and a built-in subquery statement embedded in the target aggregation operator, the built-in subquery statement including the query condition and a local aggregation operator nested outside the query condition, the local aggregation operator being used to execute the local calculation corresponding to the target aggregation operator; and generating the execution plan based on the built-in subquery statement during the calculation of the rewritten target subquery statement.

[0092] Optionally, when rewriting the target subquery statement, the processor 702 is specifically used to: rewrite the target subquery statement into the built-in subquery statement corresponding to the target aggregation operator.

[0093] Optionally, after rewriting the target subquery statement into the built-in subquery statement corresponding to the target aggregation operator, the processor 702 is further configured to: add rewriting identifier information corresponding to the built-in subquery statement to the rewritten target subquery statement; and generate an execution plan based on the built-in subquery statement during the calculation of the rewritten target subquery statement, including: parsing the rewritten target subquery statement during the calculation of the rewritten target subquery statement; and generating the execution plan based on the built-in subquery statement corresponding to the rewritten identifier information if the rewriting identifier information is parsed in the rewritten target subquery statement.

[0094] Optionally, the processor 702 is further configured to: respond to an expansion event, obtain a newly added storage node, the newly added storage node being used to receive and store newly written data together with the plurality of storage nodes; send the execution plan to the newly added storage node, the execution plan being used to enable the newly added storage node to perform a data reading operation according to the query conditions, and to perform local calculations on the read data according to the local aggregation operator.

[0095] In some exemplary embodiments, Figure 7 The illustrated electronic device is used to deploy a storage node, a processor 702 coupled to a memory 701, for executing a computer program in the memory 701 to: receive an execution plan sent by a computing node via a communication component 703, the execution plan being generated based on a built-in subquery statement of a target aggregation operator, the built-in subquery statement including: query conditions and local aggregation operators nested outside the query conditions, the local aggregation operators being used to perform local calculations corresponding to the target aggregation operator; perform a data reading operation based on data query conditions in the execution plan; perform local calculations on the read data based on the local aggregation operators to obtain local calculation results; and return the local calculation results to the computing node.

[0096] Optionally, a computing engine runs on the storage node; when the processor 702 performs local calculations on the read data according to the target aggregation operator to obtain local calculation results, it is specifically used to: call the computing engine to perform local calculations on the read data according to the local aggregation operator to obtain local calculation results.

[0097] Furthermore, such as Figure 7 As shown, the electronic device also includes other components such as a power supply component 704, a display component 705, and an audio component 706. Figure 7 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 7 The components shown. Figure 7In this embodiment, the components within the dashed boxes are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 7 The components within the dashed box; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, etc., it may be omitted. Figure 7 The component within the dashed box.

[0098] The memory 701 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0099] The communication component 703 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0100] The power supply component 704 is used to provide power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0101] The display component includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0102] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0103] In this embodiment, for the target subquery in the original query statement, multiple storage nodes can be used to perform data reading operations in parallel according to the query conditions in the target subquery. Local calculations corresponding to the target aggregation operators in the target subquery are then performed on the read data to obtain the local calculation results. Furthermore, the storage nodes no longer return the read data to the computing nodes, but rather the local calculation results of the read data. This reduces the amount of data transferred between the storage nodes and the computing nodes, thereby reducing network bandwidth consumption and computation time caused by large data transfers, and improving query efficiency. In addition, multi-node parallel computing can be implemented based on multiple storage nodes, which greatly improves computational efficiency compared to the existing single-node multi-coroutine execution method, thus adapting to massive computing scenarios.

[0104] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the server in the above method embodiments.

[0105] This application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps in the method provided in this application.

[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0112] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes said element.

[0114] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data query method applied to a computing node, wherein the data is stored in a storage layer, the storage layer comprising multiple storage nodes, characterized in that, include: Receive the calculation request corresponding to the original query statement; In the original query statement, a target subquery statement is identified, which consists of query conditions and target aggregation operators nested outside the query conditions; The target subquery statement is rewritten, and an execution plan is generated based on the rewritten target subquery statement to push down the calculation corresponding to the query condition to the storage layer. The execution plan is sent to multiple storage nodes in the storage layer. The execution plan is used to enable the multiple storage nodes to perform data reading operations in parallel according to the query conditions, and to perform local calculations corresponding to the target aggregation operator on the read data in parallel to obtain local calculation results. Based on the target aggregation operator, the local calculation results returned by the multiple storage nodes are merged to obtain the calculation result of the target sub-query statement.

2. The method according to claim 1, characterized in that, In the original query statement, the target subquery statement is identified, including: Obtain the syntax tree corresponding to the original query statement. The nodes in the syntax tree correspond to the aggregate operators in the original query statement, and the leaf nodes correspond to the query conditions in the original query statement. Traverse the nodes in the syntax tree and identify target nodes that meet set conditions during the traversal process. The set conditions include: the child nodes of the target node are leaf nodes. The target sub-query statement is determined based on the operator corresponding to the target node and the query conditions corresponding to the child nodes of the target node.

3. The method according to claim 1, characterized in that, The target subquery statement is rewritten, and an execution plan is generated based on the rewritten target subquery statement to push down the calculations corresponding to the query conditions to the storage layer, including: The target subquery statement is rewritten, and the rewritten target subquery statement includes: the target aggregation operator and the built-in subquery statement embedded in the target aggregation operator. The built-in subquery statement includes the query condition and the local aggregation operator nested outside the query condition. The local aggregation operator is used to perform the local calculation corresponding to the target aggregation operator. During the calculation of the rewritten target subquery, the execution plan is generated based on the built-in subquery.

4. The method according to claim 3, characterized in that, Rewriting the target subquery statement includes: rewriting the target subquery statement into the built-in subquery statement corresponding to the target aggregation operator.

5. The method according to claim 3, characterized in that, After rewriting the target subquery statement into the built-in subquery statement corresponding to the target aggregation operator, the method further includes: adding the rewriting identifier information corresponding to the built-in subquery statement to the rewritten target subquery statement; During the calculation of the rewritten target subquery, an execution plan is generated based on the built-in subquery, including: During the calculation of the rewritten target subquery, the rewritten target subquery is parsed. If the rewrite identifier information is parsed in the rewritten target subquery statement, then the execution plan is generated based on the built-in subquery statement corresponding to the rewrite identifier information.

6. The method according to any one of claims 3-5, characterized in that, Also includes: In response to the expansion event, the newly added storage node is obtained. The newly added storage node is used to receive and store the newly written data together with the multiple storage nodes. The execution plan is sent to the newly added storage node, which is used to enable the newly added storage node to perform data reading operations according to the query conditions and to perform local calculations on the read data according to the local aggregation operator.

7. A data query method applied to a storage node, characterized in that, include: The system receives an execution plan sent by a computing node. The execution plan is generated based on a built-in subquery statement of a target aggregation operator. The built-in subquery statement includes a query condition and a local aggregation operator nested outside the query condition. The local aggregation operator is used to perform local calculations corresponding to the target aggregation operator. Perform a data read operation based on the data query conditions in the execution plan; Based on the local aggregation operator, local calculations are performed on the read data to obtain local calculation results; The local calculation results are returned to the calculation node.

8. The method according to claim 7, characterized in that, The storage node runs a computing engine; Based on the target aggregation operator, local calculations are performed on the read data to obtain local calculation results, including: The computing engine is invoked to perform local calculations on the read data according to the local aggregation operator, and the local calculation results are obtained.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 1-6 or 7-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it can implement the data query method according to any one of claims 1-6 or 7-8.

11. A computer program product, characterized in that, include: A computer program / instruction, which, when executed by a processor, enables the data querying method according to any one of claims 1-6 or 7-8.

Images