Deep learning cost estimation system, method and device for cloud-edge collaborative query

Abstract

The invention relates to a deep learning cost estimation system, method and device for cloud-edge-end collaborative query, and belongs to the field of computer technology. In order to solve the problem that there is currently no method for optimizing joint queries of three types of devices: cloud, edge and end. In the system described in the present invention, the model input of the deep learning cost estimation model for cloud-edge-end collaborative query is determined by the system node meta-information of the cloud-edge-end system and the collaborative query plan tree, and the cost estimation model performs cost estimation based on the model input; the cost estimation model uses m tree convolution modules with residual connections to perform feature fusion and feature extraction on the collaborative query plan tree encoding, and then obtains the query plan features for each cloud / edge / end node; then GCN and tree convolution are used to efficiently fuse the query features and cloud-edge-end database system features, and accurate cost estimation of the cloud-edge-end database is achieved. The present invention is mainly used for deep learning cost estimation for cloud-edge-end collaborative query.

Description

Technical Field

[0001] The present invention relates to a cost estimation system and method for cloud-edge-terminal collaborative query, and belongs to the field of computer technology. Background Art

[0002] With the popularization of the Internet and the Internet of Things, the amount of data in edge applications has increased exponentially. In the cloud-edge-end system, data is formed and accumulated at the terminal, transmitted to the cloud, processed, and then returned to the terminal to guide the business. This series of actions will generate ultra-high demands for network bandwidth of hundreds of Gbps per second. Not only will there be delays, but there will also be many problems such as weak network freezes and low connection success rates, and the user experience cannot be guaranteed. At the same time, the large bandwidth puts huge transmission pressure on the backhaul network and business center, and also makes users face huge bandwidth costs. This means that centralized data storage, management and processing will face difficult bottlenecks and pressures. Traditional cloud computing architecture cannot meet this explosive demand for massive data computing. It will be an important development trend to sink cloud computing capabilities to the edge side and device side, and to uniformly deliver, operate, and manage through the center;

[0003] It can be said that cloud-edge-end collaboration is an extension of cloud computing, and each has its own characteristics: cloud computing can grasp the overall situation, process large amounts of data, and play an important role in non-real-time data processing scenarios such as business decision-making; cloud-edge-end collaboration can better play a role in small-scale, real-time intelligent data management, such as meeting the real-time needs of local enterprises. Therefore, cloud computing is more suitable for centralized processing of large-scale data, while cloud-edge-end collaboration can be used for small-scale intelligent data management and local services. Cloud-edge-end collaboration and cloud computing complement each other and develop in a coordinated manner, which will help the digital transformation of the industry to a greater extent.

[0004] Although cloud-edge-end collaborative data management has many advantages, due to the computing and storage resource conditions of the cloud, edge, and end and the need for collaborative data management, existing cloud database query optimization is difficult to adapt to the high-dimensional, heterogeneous data and complex architecture of cloud-edge-end database systems.

[0005] Current data management theories and technologies are difficult to meet the needs of collaborative data management between cloud, edge and end. At the same time, current research mainly focuses on data management on the cloud, while there is less research on data management on the edge and end, and even fewer on collaborative data management theories and technologies between cloud, edge and end.

[0006] In terms of end-side queries, database systems Cougar and TinyDB use extended SQL to describe queries in their designs. In terms of query optimization, their optimization methods use energy consumption and transmission costs as key indicators, and use methods such as node grouping and operator reordering as the main means of optimizing query plans.

[0007] Specifically, in the optimization schemes of the above two database systems, the database system first parses the SQL into a query plan, which specifies the calculation to be performed by each sensor (end-side device), the sampling rate, and the communication structure between sensors.

[0008] After the query plan is generated, the plan's query optimization algorithm makes query optimization decisions based on metadata, which includes static information such as sensor location, type, amount of memory, and energy consumption of each operation, as well as dynamic statistics about the remaining battery life of the sensor and the selectivity of the query predicates.

[0009] Therefore, the query optimization algorithm is based on the cost model, which is the core of the solution in the prior art. The query optimization algorithm based on the cost model takes energy consumption and transmission cost as the main consideration indicators.

[0010] Let N be the number of sensor types, k be the number of predicates, E i is the sampling cost of sensor type i, E trans 、E recv are the energy of sending and receiving a message, respectively. The energy consumed to evaluate predicate i', E agg To calculate the energy cost of (partial) aggregation, C is the number of child nodes connected by the node. The energy cost of node s collecting a sample and transmitting its partial aggregation result is estimated as follows:

[0011]

[0012] Assume that the remaining battery power of sensor s is B s Joules, s is enough power to continuously sample In order to extend the life of the system, a plan needs to be generated to maximize this value. If there are P plans, this process can be described as:

[0013]

[0014] The centralized optimizer located at the base station is responsible for determining the query plan. The optimizer has two optimization dimensions, one is the grouping of sensor nodes using a heuristic algorithm, and the other is the reordering of query operations based on heuristics.

[0015] In the field of cloud-edge-device data management, current technical solutions can only estimate the cost of device-side devices. For example, the above solution can only estimate the energy consumption of device-side devices. Cost estimation solutions in the traditional database field can only estimate the query time cost of cloud-side devices. There is no method that can connect the three types of devices, cloud, edge, and end, estimate the query time cost for the cloud-edge-end environment, and support joint query optimization in the cloud-edge-end system. Summary of the invention

[0016] The present invention aims to solve the problem that there is currently no method for joint query optimization for cloud, edge and end devices.

[0017] A deep learning cost estimation system for cloud-edge-end collaborative query, the system comprising an input data unit and a deep learning cost estimation model for cloud-edge-end collaborative query:

[0018] The input data unit determines the model input of the deep learning cost estimation model for cloud-edge collaborative query based on the metadata and collaborative logic query plan information of the cloud-edge system and the system node metadata and collaborative query plan tree of the cloud-edge system;

[0019] The deep learning cost estimation model for cloud-edge collaborative query includes a query plan feature extraction unit and a graph convolution information fusion unit;

[0020] Query feature extraction unit: receives the collaborative query plan tree code as input, uses m tree convolution modules with residual connection to perform feature fusion and feature extraction on the collaborative query plan tree code; extracts a 1×F ′ n After the eigenvector is obtained, the vector is replicated N times along axis = 0 to form N vectors with a length of 1×F ′ n The i-th feature vector corresponds to a cloud / edge / end physical node i in the cloud-edge-end system. Then, for each cloud / edge / end physical node i, according to the database relation operation op actually performed by it, the part of the i-th feature vector corresponding to the operation op is retained, and the rest is set to 0 using the mask operation, thus obtaining the final query plan feature F' for each cloud / edge / end node.

[0021] Graph convolution information fusion unit: receives the corresponding N×N adjacency matrix and N×F feature matrix in the system node meta-information, which are recorded as adjacency matrix N and feature matrix F respectively;

[0022] The query plan features F' and the feature matrix F are then concatenated into a fused feature matrix F'-F, and a graph convolutional neural network based on the spectral method is used to extract features from the adjacency matrix N and the fused feature matrix F'-F. Finally, the extracted results are expanded into a one-dimensional vector by row, and the final cost estimation result is output after passing through two layers of fully connected neural networks.

[0023] Furthermore, the system node meta information includes system node topology information and metadata characteristics of each node;

[0024] The system node topology information is regarded as an undirected graph and encoded as an N×N adjacency matrix, where N is the total number of nodes corresponding to the cloud, edge, and end, and the elements in the matrix represent the network bandwidth between two nodes in Gbps;

[0025] The metadata feature of each node is an N×F matrix, where F is the metadata dimension. The metadata of a single node includes static metadata, node computing load, and node I / O load.

[0026] The static metadata includes: node type, node model, node remaining power, energy consumption cost of node performing data operations such as sampling, filtering, aggregation, selection, etc., node computing power, and node storage capacity.

[0027] Furthermore, the node type and node model in the static metadata are set in one-hot encoding form; the remaining power of the node is only valid for the end node, and this value is 0 for other types of nodes; the energy consumption cost of the node performing data operations such as sampling, filtering, aggregation, and selection is only valid for the end node, and this value is 0 for other types of nodes.

[0028] Furthermore, the collaborative query plan tree is divided into two parts: topology structure and node features for encoding:

[0029] The collaborative query plan tree is divided into a binary tree structure with topological structure, and the node codes are sequentially spliced ​​according to the tree level traversal order;

[0030] The node feature of the node feature is composed of the encoding of the node information; the node information includes: the one-hot encoding of the node type, the table involved in the operation, the index type involved in the operation, the predicate involved in the operation, and the cardinality estimation result of the node.

[0031] Furthermore, the encoding of the node information is as follows:

[0032] The node type is one-hot encoded; the table involved in the operation is one-hot encoded; the index type involved in the operation is one-hot encoded; the predicate involved in the operation is one-hot encoded; the cardinality estimation result of the node is an integer value.

[0033] The deep learning cost estimation method based on cloud-edge collaborative query includes the following steps:

[0034] After obtaining the user's query statement, the SQL query statement input by the user is parsed into a query plan tree using database technology, and a vector tree encoding of the collaborative query plan tree is generated;

[0035] Then analyze the physical nodes of each operator in the query plan tree and generate a Mask matrix;

[0036] Predicting the cost of the query based on the deep learning cost estimation system for cloud-edge collaborative query;

[0037] Perform actual queries and generate actual costs; the cloud-edge system collects the actual costs of the queries and saves data pairs in the database; the data pairs refer to the data pairs consisting of the model input and actual costs of the deep learning cost estimation model for cloud-edge collaborative queries.

[0038] Furthermore, the estimation method further comprises the following steps:

[0039] When the number of new data entries in the database exceeds a manually preset threshold, the new data is used to update the cost estimation model through incremental training.

[0040] Furthermore, in the process of cost estimation, it is necessary to regularly extract system meta-information of the cloud-edge system, where the system meta-information includes system node meta-information and collaborative query plan tree information.

[0041] A deep learning cost estimation device for cloud-edge collaborative query, the device is a storage medium, the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the deep learning cost estimation system for cloud-edge collaborative query.

[0042] A deep learning cost estimation device for cloud-edge collaborative query, the device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the deep learning cost estimation system for cloud-edge collaborative query.

[0043] Beneficial effects:

[0044] Different from the existing cloud database cost estimation and edge database energy consumption technology, this paper takes the lead in designing a cost estimation model for collaborative queries on cloud-edge-end databases, and innovatively proposes to use GCN and tree convolution to efficiently integrate query features and cloud-edge-end database system features, thus achieving accurate cost estimation for cloud-edge-end databases. Therefore, this paper can optimize joint queries on cloud, edge, and end devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the adjacency matrix of three nodes;

[0046] Figure 2 Encoding examples for node data features;

[0047] Figure 3 Coding examples for query plans;

[0048] Figure 4It is a schematic diagram of the model structure of the present invention. DETAILED DESCRIPTION

[0049] The purpose of the present invention is to design a deep learning-driven, cloud-edge-end collaborative query cost estimation model. The model uses advanced deep learning technologies such as GCN and Attention, and uses information such as system metadata, query plan embedding coding, and data distribution metadata to accurately estimate the query costs of cloud, edge, and end devices, thereby providing a basis for cloud-edge-end collaborative query optimization methods based on cost estimation. The present invention fills the technical gap in the classic problem of query cost estimation in the database field in the cloud-edge-end collaborative scenario. Specific implementation method one:

[0051] The deep learning cost estimation method for cloud-edge-device collaborative query described in this embodiment includes the following steps:

[0052] A deep learning cost estimation model for cloud-edge-device collaborative query is constructed. The deep learning cost estimation model for cloud-edge-device collaborative query is as follows:

[0053] 1. Model input:

[0054] The present invention makes full use of the metadata and collaborative logical query plan information of the cloud-edge system, and uses the system node meta-information (system node topology information + metadata features of each node) and the collaborative query plan tree as the model input, where:

[0055] (1) The system node topology information is regarded as an undirected graph and encoded as an N×N adjacency matrix, where N is the total number of nodes corresponding to the cloud, edge, and end, and the elements in the matrix represent the network bandwidth between two nodes in Gbps; Figure 1 As shown, Figure 1 It is a schematic diagram of the adjacency matrix of three nodes;

[0056] (2) The metadata feature of each node is an N×F matrix, where Ff is the metadata dimension. The metadata of a single node includes static metadata, as well as node computing load (i.e., the node's current CPU usage) and node I / O load (node's current I / O throughput / node's maximum I / O throughput). The node data feature encoding example is as follows: Figure 2 shown.

[0057] The static metadata includes:

[0058] Node type (one-hot encoding, 100: cloud node, 010 edge node, 001 end node);

[0059] Node model (using one-hot encoding, length f', which can be adjusted according to the number of node models in the system);

[0060] The remaining power of the node (only valid for peer nodes, the value is 0 for other types of nodes);

[0061] The energy cost of the node performing data operations such as sampling, filtering, aggregation, and selection (only valid for peer nodes, the value is 0 for other types of nodes);

[0062] Node computing power (in Gflops);

[0063] Node storage capacity (in GB, this value is set to 0 for end nodes without storage capacity);

[0064] (3) The collaborative query plan tree is similar to the query plan tree in a relational database. It is encoded in two parts: topology structure and node features:

[0065] Topological structure: binary tree structure, each node code is concatenated in the order of tree layer traversal;

[0066] Node characteristics: 1×f n One-dimensional vector of node features; 1×f n A one-dimensional vector is constructed by the following encoding:

[0067] One-hot encoding of node types (including empty nodes);

[0068] The table involved in the operation (one-hot encoding);

[0069] The type of index involved in the operation (one-hot encoding);

[0070] The predicate involved in the operation (one-hot encoding);

[0071] The cardinality estimation result of the node (integer value);

[0072] The query plan encoding example is as follows Figure 3 shown.

[0073] (II) Model structure:

[0074] In order to use system meta-information as semi-supervised information to assist in predicting the cost of queries, the model architecture adopted by the present invention is divided into two parts: a query plan feature extraction unit and a graph convolution information fusion unit;

[0075] Query feature extraction unit: receives the collaborative query plan tree encoding as input, uses m tree convolution modules with residual connection to perform feature fusion and feature extraction on the collaborative query plan tree encoding; extracts a 1×F′ n After the eigenvector is obtained, the vector is replicated N times along axis = 0 to form N vectors with a length of 1×F′ nThe i-th feature vector corresponds to a cloud / edge / end physical node i in the cloud-edge-end system; then, for each cloud / edge / end physical node i, according to the database relational operation op actually performed by it (i.e., sequential scanning, merge join and other operations in the vector tree), the part of the i-th feature vector corresponding to the operation op is retained, and the rest is set to 0 using the mask operation to obtain the final query plan feature F' for each cloud / edge / end node.

[0076] Graph convolution information fusion unit: receives the features corresponding to the system node meta-information and the query plan features F'. The system node meta-information is represented as an N×N adjacency matrix and an N×F feature matrix, which are respectively recorded as the adjacency matrix N and the feature matrix F;

[0077] Then, the query plan features F' and the feature matrix F obtained previously are concatenated into a fused feature matrix F'-F, and then a graph convolutional neural network based on a spectral method (GCN based on a spectral method) is used to extract features from the adjacency matrix N and the fused feature matrix F'-F. Finally, the extracted results are expanded into a one-dimensional vector by row, and the final cost estimation result is output after passing through two layers of fully connected neural networks.

[0078] The process of cost estimation using the deep learning cost estimation model for cloud-edge collaborative query includes the following steps:

[0079] 0. Regularly extract cloud-edge system metadata:

[0080] Since the cloud-edge system meta-information contains dynamic features such as the current node load, the cloud-edge system will regularly update the system meta-information according to the preset time interval; the system meta-information includes system node meta-information and collaborative query plan tree information.

[0081] The present invention records it as step 0 because the periodic extraction of system meta information is regularly updated according to a preset time interval and is the basis for subsequent steps;

[0082] 1. After the user inputs the query statement, the existing database technology is used to parse the SQL query statement entered by the user into a query plan tree and generate the following Figure 3 The vector tree encoding of the collaborative query plan tree shown;

[0083] 2. Analyze the physical nodes of each operator in the query plan tree and generate a Mask matrix;

[0084] 3. Use the tree convolution module and the mask matrix to extract the features of the query plan tree;

[0085] 4. Concatenate the features extracted in step 3 with the feature matrix in the system metadata;

[0086] 5. Use graph convolutional neural network based on spectral method to extract graph features;

[0087] 6. Use the fully connected network to finally predict the cost of the query;

[0088] 7. Perform actual query and generate actual cost;

[0089] 8. The cloud-edge system collects the actual cost of the query and saves the data pair (model input and actual cost) to the database;

[0090] 9. When the number of new data entries in the database exceeds the manually preset threshold, the new data is used to update the cost estimation model through incremental training based on the current cost estimation model. Specific implementation method 2:

[0092] This embodiment is a deep learning cost estimation device for cloud-edge collaborative query. The device is a storage medium, and the storage medium stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the estimation process of the deep learning cost estimation model for cloud-edge collaborative query.

[0093] The at least one instruction can also be loaded and executed by the processor to obtain or generate a process for deep learning cost estimation model input for cloud-edge collaborative queries. Specific implementation method three:

[0095] This embodiment is a deep learning cost estimation device for cloud-edge collaborative query, the device includes a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the estimation process of the deep learning cost estimation model for cloud-edge collaborative query.

[0096] The at least one instruction can also be loaded and executed by the processor to obtain or generate a process for deep learning cost estimation model input for cloud-edge collaborative queries.

[0097] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A deep learning cost estimation system for cloud-edge collaborative query, characterized by: The system includes an input data unit and a deep learning cost estimation model for cloud-edge collaborative query: The input data unit determines the model input of the deep learning cost estimation model for cloud-edge collaborative query based on the metadata and collaborative logic query plan information of the cloud-edge system and the system node metadata and collaborative query plan tree of the cloud-edge system; The deep learning cost estimation model for cloud-edge collaborative query includes a query plan feature extraction unit and a graph convolution information fusion unit; Query feature extraction unit: receives the collaborative query plan tree code as input, uses m tree convolution modules with residual connection to perform feature fusion and feature extraction on the collaborative query plan tree code; extracts the length of After the eigenvector is obtained, the vector is replicated N times along axis=0 to form N vectors with a length of The i-th feature vector corresponds to a cloud / edge / end physical node i in the cloud-edge-end system. Then, for each cloud / edge / end physical node i, according to the database relation operation op actually performed by it, op refers to the operation in the vector tree, the part of the i-th feature vector corresponding to the operation op is retained, and the rest is set to 0 using the mask operation, and the final query plan feature F' for each cloud / edge / end node is obtained. Graph convolution information fusion unit: receives the corresponding system node meta information The adjacency matrix and The feature matrices are denoted as adjacency matrix N and feature matrix F respectively; The query plan features F' and the feature matrix F are then concatenated into a fused feature matrix F'-F, and a graph convolutional neural network based on the spectral method is used to extract features from the adjacency matrix N and the fused feature matrix F'-F. Finally, the extracted results are expanded into a one-dimensional vector by row, and the final cost estimation result is output after passing through two layers of fully connected neural networks.

2. The deep learning cost estimation system for cloud-edge collaborative query according to claim 1 is characterized in that: The system node meta information includes system node topology information and metadata characteristics of each node; The system node topology information is viewed as an undirected graph and encoded as a The adjacency matrix of is the total number of nodes corresponding to the cloud, edge, and end. The elements in the matrix represent the network bandwidth between two nodes in Gbps; Each node metadata feature is a The matrix of The metadata dimension is the metadata of a single node, including static metadata, node computing load, and node I / O load. The static metadata includes: node type, node model, node remaining power, energy consumption cost of node execution of sampling, filtering, aggregation, and data selection operations, node computing power, and node storage capacity.

3. The deep learning cost estimation system for cloud-edge collaborative query according to claim 2 is characterized in that: The node type and node model in the static metadata are set in one-hot encoding form; the remaining power of the node is only valid for the end node, and this value is 0 for other types of nodes; the energy consumption cost of the node performing sampling, filtering, aggregation, and data selection operations is only valid for the end node, and this value is 0 for other types of nodes.

4. The deep learning cost estimation system for cloud-edge collaborative query according to claim 2 or 3, characterized in that: The collaborative query plan tree is divided into two parts: topology structure and node features for encoding: The collaborative query plan tree is divided into a binary tree structure with topological structure, and the node codes are sequentially spliced ​​according to the tree level traversal order; The node feature of the node feature is composed of the encoding of the node information; the node information includes: the one-hot encoding of the node type, the table involved in the operation, the index type involved in the operation, the predicate involved in the operation, and the cardinality estimation result of the node.

5. The deep learning cost estimation system for cloud-edge collaborative query according to claim 4 is characterized in that: The encoding of the node information is as follows: The node type is one-hot encoded; the table involved in the operation is one-hot encoded; the index type involved in the operation is one-hot encoded; the predicate involved in the operation is one-hot encoded; the cardinality estimation result of the node is an integer value.

6. A deep learning cost estimation method based on cloud-edge collaborative query, characterized in that: The following steps are involved: After obtaining the user's query statement, the SQL query statement input by the user is parsed into a query plan tree using database technology, and a vector tree encoding of the collaborative query plan tree is generated; Then analyze the physical nodes of each operator in the query plan tree and generate a Mask matrix; Predicting the cost of a query based on the deep learning cost estimation system for cloud-edge collaborative query according to any one of claims 1 to 5; Perform actual queries and generate actual costs; the cloud-edge system collects the actual costs of the queries and saves data pairs in the database; the data pairs refer to the data pairs consisting of the model input and actual costs of the deep learning cost estimation model for cloud-edge collaborative queries.

7. The deep learning cost estimation method based on cloud-edge-device collaborative query according to claim 6 is characterized in that: The estimation method further comprises the following steps: When the number of new data entries in the database exceeds a manually preset threshold, the new data is used to update the cost estimation model through incremental training.

8. The deep learning cost estimation method based on cloud-edge-device collaborative query according to claim 7 is characterized in that: In the process of cost estimation, it is necessary to regularly extract system meta-information of the cloud-edge system, where the system meta-information includes system node meta-information and collaborative query plan tree information.

9. A deep learning cost estimation device for cloud-edge collaborative query, characterized in that: The device is a storage medium in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the deep learning cost estimation system for cloud-edge collaborative query as described in one of claims 1 to 5.

10. A deep learning cost estimation device for cloud-edge collaborative query, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the deep learning cost estimation system for cloud-edge collaborative query as described in one of claims 1 to 5.

Images