Automated query selection prediction using query graphs

Abstract

Examples described herein provide a computer-implemented method that includes training a machine learning model. The model is trained by: generating a training query set using at least one of query workloads and relationships between tables in a database; constructing a query graph for each training query in the training query set; calculating, for each training query in the set of training queries, a selection based at least in part on the query graph; and constructing an initial connection result distribution as a query graph set based at least in part on the training query set.

Description

Background Art

[0001] Embodiments described herein relate generally to processing systems and, more particularly, to automated query selectivity prediction using query graphs.

[0002] Query optimization is the process of creating a management system in a relational database to efficiently execute queries.

[0003] Structured Query Language (SQL) is a programming language created and designed to manage data stored in a relational database. Structured Query Language (SQL) may include a specific programming language for managing data stored in a relational database management system (RDBMS).

[0004] A relational database management system (RDBMS) typically may include single column statistics that are usually collected for individual columns in a relation. A relation may include tuples or attributes that describe a relationship or define characteristics or relationships between tables. For example, a relation includes data values ​​about a table, and a relational database may store these data values ​​as a relation or table. A collection of relations or tables may be stored in a database as a relational model. Summary of the invention

[0005] In an exemplary embodiment, a computer-implemented method is provided. The method includes training a machine learning model. The model is trained by: generating a training query set using at least one of a query workload and a relationship between tables in a database; for each training query in the training query set, building a query graph; for each training query in the training query set, calculating a selectivity based at least in part on the query graph; and building an initial join result distribution as a query graph set based at least in part on the training query set.

[0006] In another exemplary embodiment, a computer-implemented method is provided. The method includes training a machine learning model. The model is trained by the following steps: generating a training query set using at least one of a query workload and a relationship between tables in a database; for each training query in the training query set, building a query graph; for each training query in the training query set, calculating selectivity based at least in part on the query graph; and building an initial connection result distribution as a query graph set based at least in part on the training query set. The method also includes: after training the machine learning model, using the machine learning model to perform reasoning. Performing reasoning includes: calculating a connection cardinality for each connection statement of the query; and identifying the best fit in the query graph set to generate an associated cardinality estimate. The method also includes: performing continuous training on the machine learning model. Performing continuous training includes: editing new queries in a new query workload based on workload, new tables, and changes to data using inputs including at least one change selected from a group including new queries; and updating the query graph set.

[0007] Other embodiments described herein implement the features of the above methods in computer systems and computer program products.

[0008] The above features and advantages and other features and advantages of the present disclosure will be readily apparent from the following detailed description when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The details of the exclusive rights described herein are particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of the embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 depicts a block diagram of a processing system for implementing one or more embodiments described herein;

[0011] Figure 2 depicts a block diagram of components of a machine learning training and inference system according to one or more embodiments described herein;

[0012] Figure 3 Depicted is a method for estimating the cardinality of a query using a query graph in accordance with one or more embodiments described herein;

[0013] Figure 4 depicts a query graph according to one or more embodiments described herein;

[0014] Figure 5depicts a spatial diagram showing two tables with their join columns according to one or more embodiments described herein;

[0015] Figure 6 Depicts the features of each node representing the initial coordinates in the graph space;

[0016] Figure 7 depicts a result query graph with optimized node positioning for three-table queries according to one or more embodiments described herein;

[0017] Figure 8 depicts a resulting query graph with updated positions after optimization according to one or more embodiments described herein;

[0018] Fig. 9A , 9B 9C together depict an example of constructing a query graph for a given query according to one or more embodiments described herein; and

[0019] Fig.10 depicts a flow chart of a method for training a machine learning model to estimate the cardinality of a query using a query graph in accordance with one or more embodiments described herein;

[0020] The figures depicted herein are illustrative. Without departing from the scope of the present invention, many changes may be made to the figures or operations described therein. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. In addition, the term "coupled" and its variations describe having a communication path between two elements and do not mean a direct connection between elements without intermediate elements / connections. All of these changes are considered part of the specification. DETAILED DESCRIPTION

[0021] One or more embodiments described herein provide for estimating the cardinality of a query using a query graph.

[0022] Various aspects of the present invention are described by explanatory text, flow charts, block diagrams of computer systems, and / or block diagrams of machine logic, included in computer program product (CPP) embodiments. For any flow chart, depending on the technology involved, the operations may be performed in an order different from the order shown in a given flow chart. For example, two operations shown in consecutive flow chart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time, also depending on the technology involved.

[0023] Computer program product embodiments ("CPP embodiments" or "CPP") are terms used in this disclosure to describe any collection of one or more storage media (also referred to as "media") that are collectively included in a collection of one or more storage devices that collectively include machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can retain and store instructions for use by a computer processor. A computer-readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the above. Some known types of storage devices that include these media include: floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device (such as a punch card or pits / islands formed in a major surface of a disk), or any suitable combination of the above. As the term is used in the present invention, computer readable storage media is not to be interpreted as storage in the form of transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, light pulses through optical cables, electrical signals transmitted through wires and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at certain occasional points in time during normal operation of the storage device, such as during access, defragmentation or garbage collection, but this does not make the storage device temporary because the data is not temporary when stored.

[0024] The computing environment 100 includes an example of an environment for executing at least some of the computer codes involved in performing the methods of the present invention, such as using a query graph to estimate the cardinality of a query 150. In addition to the block 150, the computing environment 100 also includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor group 110 (including a processing circuit 120 and a cache 121), a communication structure 111, a volatile memory 112, a persistent storage device 113 (including an operating system 122 and a block 150, as described above), a peripheral device group 114 (including a user interface (UI) device group 123, a storage device 124, and an Internet of Things (IoT) sensor group 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140 , a cloud orchestration module 141 , a host physical machine group 142 , a virtual machine group 143 , and a container group 144 .

[0025] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smart phone, a smart watch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or developed in the future that is capable of running programs, accessing a network, or querying a database, such as remote database 130. As is understood in the art of computer technology, execution of computer-implemented methods may be distributed among multiple computers and / or among multiple locations. On the other hand, in the presentation of this computing environment 100, the detailed discussion focuses on a single computer, particularly computer 101, to keep the presentation as simple as possible. Computer 101 may be located in the cloud, although Figure 1 On the other hand, computer 101 need not be in the cloud unless it can be positively indicated to any extent.

[0026] Processor group 110 includes one or more computer processors of any type now known or to be developed in the future. Processing circuit 120 can be distributed over multiple packages, such as multiple coordinated integrated circuit chips. Processing circuit 120 can implement multiple processor threads and / or multiple processor cores. Cache 121 is a memory located in the processor chip package, typically used for data or code that threads or cores running on processor group 110 should have quick access to. Cache memories are typically organized into multiple levels based on relative proximity to the processing circuit. Alternatively, some or all of the caches of the processor group may be located "off-chip". In some computing environments, processor group 110 may be designed to use quantum bits and perform quantum computing.

[0027] Computer readable program instructions are typically loaded onto the computer 101 to cause the processor group 110 of the computer 101 to execute a series of operating steps to implement a computer-implemented method, such that the instructions so executed will instantiate the method specified in the flowchart and / or illustrative description of the computer-implemented method contained in this document (collectively referred to as the "method of the present invention"). These computer readable program instructions are stored in various types of computer readable storage media, such as cache 121 and other storage media discussed below. The program instructions and related data are accessed by the processor group 110 to control and directly execute the method of the present invention. In the computing environment 100, at least some of the instructions for executing the method of the present invention may be stored in a block 150 in the persistent storage device 113.

[0028] The communication fabric 111 is the signaling pathways that allow the various components of the computer 101 to communicate with each other. Typically, the fabric is comprised of switches and conductive pathways, such as those that make up a bus, a bridge, physical input / output ports, etc. Other types of signal communication pathways may be used, such as fiber optic communication pathways and / or wireless communication pathways.

[0029] The volatile memory 112 is any type of volatile memory currently known or developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, volatile memory is characterized by random access, but this is not required unless explicitly indicated. In the computer 101, the volatile memory 112 is located in a single package and is internal to the computer 101, but, alternatively or additionally, the volatile memory can be distributed over multiple packages and / or located externally relative to the computer 101.

[0030] Persistent storage 113 is any form of non-volatile storage for a computer known now or to be developed in the future. The non-volatility of the storage means that the stored data will be maintained regardless of whether power is supplied to computer 101 and / or whether power is supplied directly to persistent storage 113. Persistent storage 113 may be a read-only memory (ROM), but typically at least a portion of the persistent storage allows writing of data, deletion of data, and rewriting of data. Some common forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take a variety of forms, such as various known proprietary operating systems or open source portable operating system interface type operating systems using a kernel. The code included in block 150 typically includes at least some computer code involved in executing the method of the present invention.

[0031] The peripheral device group 114 includes a peripheral device group of the computer 101. The data communication connection between the peripheral device and other components of the computer 101 can be implemented in various ways, such as a Bluetooth connection, a near field communication (NFC) connection, a cable connection (such as a universal serial bus (USB) type cable), a plug-in connection (such as a secure digital (SD) card), a connection through a local area communication network, or even a connection through a wide area network such as the Internet. In various embodiments, the UI device group 123 may include components such as a display screen, a speaker, a microphone, a wearable device (such as goggles and smart watches), a keyboard, a mouse, a printer, a touchpad, a game controller, and a tactile device. The storage device 124 is an external memory (such as an external hard drive) or a pluggable storage device (such as an SD card). The storage device 124 can be persistent and / or volatile. In some embodiments, the storage device 124 can take the form of a quantum computing storage device for storing data in the form of quantum bits. In embodiments where computer 101 is required to have a large amount of storage (e.g., computer 101 locally stores and manages a large database), the storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically dispersed computers. IoT sensor group 125 includes sensors that can be used in IoT applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0032] The network module 115 is a collection of computer software, hardware, and firmware that allows the computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware, such as a modem or a Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for transmission over a communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control function and the network forwarding function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software defined networks (SDN)), the control function and forwarding function of the network module 115 are executed on physically separated devices, so that the control function manages multiple different network hardware devices. Computer-readable program instructions for executing the method of the present invention can typically be downloaded to the computer 101 from an external computer or an external storage device via a network adapter card or a network interface contained in the network module 115.

[0033] WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any technology for transmitting computer data now known or developed in the future. In some embodiments, the WAN may be replaced and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area such as a Wi-Fi network. A WAN and / or LAN typically includes computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0034] End-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of an enterprise operating computer 101), and may take any of the forms discussed above in connection with computer 101. EUD 103 typically receives data that is helpful and useful from the operation of computer 101. For example, in the hypothetical case where computer 101 is designed to provide recommendations to an end-user, the recommendations would typically be transmitted from network module 115 of computer 101 to EUD 103 via WAN 102. In this way, EUD 103 may display or otherwise present the recommendations to the end-user. In some embodiments, EUD 103 may be a client device, such as a thin client, a heavy client, a mainframe, a desktop, etc.

[0035] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, the historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0036] The public cloud 105 is any computer system that is available to multiple entities and provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing capabilities, without requiring direct active management by users. Cloud computing typically utilizes resource sharing to achieve consistency and economies of scale. Direct active management of the computing resources of the public cloud 105 is performed by computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that constitute the host physical machine group 142, which are various physical computers in and / or available for the public cloud 105. The virtual computing environment (VCE) is typically in the form of a virtual machine from the virtual machine group 143 and / or a container from the container group 144. It will be appreciated that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after the VCE is instantiated. The cloud orchestration module 141 manages the transmission and storage of images, deploys new instances of the VCE, and manages active instances of VCE deployment. Gateway 140 is a collection of computer software, hardware, and firmware that allows public cloud 105 to communicate over WAN 102 .

[0037] Some further explanation of virtualized computing environments (VCEs) will now be provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two familiar types of VCEs are virtual machines and containers. Containers are VCEs that use operating system level virtualization. This refers to an operating system feature where the kernel allows multiple isolated user space instances (called containers) to exist. From the perspective of the programs running in them, these isolated user space instances typically behave like real computers. Computer programs running on a normal operating system can utilize all of the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running within a container can only use the contents of that container and the devices assigned to that container, a feature known as containerization.

[0038] Private cloud 106 is similar to public cloud 105, except that the computing resources are only available to a single enterprise. Although private cloud 106 is depicted as communicating with WAN 102, in other embodiments, the private cloud can be completely disconnected from the Internet and can only be accessed through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), typically implemented separately by different vendors. Each of the multiple clouds remains an independent discrete entity, but the larger hybrid cloud architecture is tied together through standardized or proprietary technologies that support orchestration, management, and / or data / application portability between multiple constituent clouds. In this embodiment, both public cloud 105 and private cloud 106 are part of a larger hybrid cloud.

[0039] The computing environment 100 may access or manage a database such as a remote database 130. In some embodiments, the remote database 130 may implement a relational model that may be based on predicate logic that may include conditions to be applied to a table or relationship. In a relational model, if statistics are not collected about multiple columns in a table, the predicates may be assumed to be independent when calculating cardinality. Predicates may typically be expressed in the form of ranges that require lower and upper limits. BETWEEN predicates and range predicates may be formed as separate value ranges for each predicate. Equality predicates may have equal lower and upper limits. This may provide some improvements in cardinality estimates; however, the computational power to implement improved cardinality estimates is extensive, expensive, and time intensive.

[0040] Cardinality estimation plays an important role in Structured Query Language (SQL) optimization. Cardinality as it relates to Structured Query Language (SQL) refers to the number of rows processed by the various operators in a query execution plan. As the number of rows is processed by the different operators, the number of rows entering the operators may decrease or increase until the final result is returned to the application or to the user who issued the query. This is different from column cardinality. Column cardinality is a count of the number of distinct values ​​in a column of a database table. A scan of a database table may apply a predicate that reads all rows in the table; however, the output may only include rows that meet the predicate conditions. Cardinality estimation is a prediction of how many rows meet the conditions versus how many rows are input to the scan operator.

[0041] A cost-based optimizer may rely on cardinality estimates to calculate access plan execution costs and select an optimal access plan. A cost-based optimizer may use statistics based on table data values ​​in a database table to identify efficient plans for query execution for a structured query. Therefore, the accuracy of the cardinality of a structured query may have an impact on the cost-based optimizer. The accuracy of the cardinality contributes to the overall goal of selecting an optimal execution plan through cardinality estimates. Accuracy in cardinality estimates may be difficult, or the information required to make the process easier may be expensive to collect, or both. Accuracy in cardinality is particularly difficult to achieve on tables with correlated or skewed columns. Inaccurate cardinality estimates may result in selection of slow access plans, unpredictable query performance, slow or unstable query performance, etc., including combinations and / or multiple effects thereof.

[0042] Query optimization relies on the evaluation of query access plans, which predict the optimal access of each table in the query and the join method and join order based on some cost estimates. The selection of the optimal query access plan is highly dependent on the accuracy of the cost estimates of the operators. Cardinality estimates are an element of predicting such costs. In some cases, optimization depends on the accuracy of such estimates. The application of predicates in a query affects the cardinality estimates and can be divided into two aspects: estimating the cardinality after applying local predicates to the base tables, and estimating the cardinality of joins. Performing these estimates is usually a difficult task requiring some complexity, requiring consideration of consistency, skewness, relevance, and predicate types (e.g., equality, range, negation, conjunction, disjunction, etc., including combinations and / or multiple forms thereof). Inaccuracies in such estimates lead to suboptimal query access plan selection, which results in slow or unpredictable query performance.

[0043] Traditional optimizer techniques for optimizing query execution rely on some limited collected statistics. The collected statistics decompose large distributions with multiple columns into fewer parameters describing the number of distinct values, the top n frequency values, and histograms of individual columns. In these cases, independence between column values ​​for columns involved in multiple predicates is assumed and is referred to as the independence assumption. The independence assumption is useful for overcoming lost relevant information. However, if individual columns are correlated, estimation errors about local predicates propagate through joins and accumulate. For example, using statistical views to collect column-group statistics or collecting detailed statistics about join results can help solve the estimation problem, but at the cost of very high tuning costs. It is difficult to determine which columns are potentially useful for statistically grouping together. It is also difficult to continuously collect and store this data frequently as the table data changes. For example, for every n columns, 2 n-1 groups to cover all possible combinations.

[0044] Tuning to obtain better query execution plans is often difficult because it relies on deep expertise. Conventional methods for improving cardinality estimates range from manually running advisors to collect high-level statistics to just-in-time targeted statistics collection on a sample of data prior to query optimization. Running statistics collection can add performance overhead. In some cases, automatic column group statistics (CGS) collection done by a background daemon has been attempted, which can help obtain better estimates for more than statistically related columns with multiple predicates. In this case, the daemon wakes up periodically (every 2 hours); however, this may be too late because queries that need this information may be running. In addition, not all problems are solved with CGS, such as skew or mismatch in the join column value domain, where minimal statistics may not be able to overcome the assumptions made when computing the join predicate cardinality estimates.

[0045] Relying on certain assumptions to predict the joint distribution of multiple relations often leads to significant cardinality estimation errors. Independence, consistency, and inclusion assumptions are just three examples of causes of these errors. This can lead to very poor query performance. In addition, sampling on join columns can quickly end up with the known problem of zero-tuple join results (e.g., collecting statistics by joining independently collected samples on join predicate columns that are uncorrelated).

[0046] Efforts have been made to address some of these problems using machine learning (ML). ML has shown the potential to outperform statistical-based estimation functions to estimate predicate / join / query cardinality. Learning models can achieve better performance than statistical-based methods.

[0047] One such approach proposes a learning model that learns single and joined cross tables. This approach captures related columns in single or multiple tables. The learning model used is a deep learning neural network built in the form of multiple sets of convolutional networks. This learning model encodes each query using materialized samples. Each query is expressed as features in terms of three main sets. This approach provides an estimate of the cardinality of the entire query. The query is executed against the database to obtain the actual cardinality. However, this approach is highly dependent on materialized samples from each table in the database and requires identification of columns and tables through encoding. This makes the training very specific to named tables and columns and cannot be generalized to unseen tables or columns.

[0048] Another approach samples the data to reduce the overhead of estimating range predicates and approximate the join data distribution. However, sampling the data for each range predicate on the fly and feeding them to the model to estimate their density to sum the results is very expensive in terms of processing and memory load and is not practical.

[0049] Another approach proposes to use machine learning to estimate query cardinality. This approach relies on statistics from previously executed queries. The actual cardinality is learned from the query structure, where the predicates are represented by participating columns and associated operators. However, this approach assumes that the workload is static and there is only one correct answer for each query structure.

[0050] Another approach proposes to use machine learning for cardinality estimation instead of using basic neural networks, recurrent neural networks, and random forests. The proposed model uses single-dimensional vectors: relations, predicates, and join predicates. Literals found in the query are normalized to values ​​between 0 and 1. Join predicates are encoded using one-hot encoding. The overall goal is to let the model learn the selectivity of the join operation. However, when building the model, this approach incurs considerable overhead since the construction of the model is computationally expensive and time-consuming in terms of processing and memory load.

[0051] Another approach proposes to create smaller models on a subset of connections. In this approach, only 2-way and 3-way connections are trained for these specific connections, thereby capturing different combinations of local predicates. This approach has not yet been generalized to arbitrary 2-, 3-, or n-way connections.

[0052] Yet another approach proposes using deep neural networks to build query execution plans by continuously learning from previously executed queries. This approach learns correlations through word similarities in the data, which can produce a model. This approach has limited generalization capabilities to columns not seen during training.

[0053] To overcome these and other shortcomings, one or more embodiments described herein provide for learning the selectivity of a full query using a machine learning model. The learned selectivity of the full query can then be used to predict a more accurate cardinality estimate for any access plan, for a sub-partial plan, and / or for any query that applies a join predicate. The ML model learns from the training workload and improves its predictions accordingly. One or more embodiments described herein provide improved robustness of learning from several examples represented by different distributions and generalizing predictions on unseen tables / queries. The method is capable of handling multiple tables with multiple join columns and multiple local predicates.

[0054] One or more embodiments described herein use machine learning to approximate the cardinality of multivariate result queries after applying local and join predicates. To achieve this, a query graph representation of the query attributes is generated. The query graph representation (or simply "query graph") is an n-dimensional representation of nodes. The nodes represent relational database tables. By representing queries in the form of a query graph, a machine learning model can learn the queries and generate associated prediction cardinality. The resulting query graph is robust for processing local and join predicates, and is normalized to represent tables and connections with attributes so that it does not depend on table names and column names. One or more embodiments described herein provide retraining, for example, by providing the ability to learn incrementally as new tables and workloads are added. One or more embodiments described herein provide retraining, for example, by running a cumulative training workload for full model retraining.

[0055] One or more embodiments of learning join cardinalities described herein provide one or more advantages over existing machine learning models. One or more embodiments described herein provide the ability to learn the distribution of join results. For example, one or more embodiments enable a learning model to understand the distribution properties of multiple relations in a given query graph, avoiding the need to utilize multi-column statistics or collect statistical views or materialized query tables (also known as materialized views), and / or provide further benefits of pairwise join correlation with any local predicates and with other pairwise joins.

[0056] One or more embodiments described herein provide for generalization. For example, one or more embodiments using query graphs provide for avoiding encoding of tables, connections, or columns with local or join predicates. In this way, the model learns various properties of queries connecting multiple relations while generalizing to newly created tables or new queries on unseen join columns. Additionally or alternatively, if the underlying distribution of relations has changed over time, the learned model does not associate the encoding of the specific tables, columns, and connections used during training, but rather associates the properties of the new distribution.

[0057] One or more embodiments described herein provide feature decomposition. For example, a query graph provides a nomenclature that defines how to draw nodes in a multidimensional graph (query graph). The query graph provides queries to be encoded with multiple features, rather than arranging the queries in sequence as input to the model, which would significantly increase the number of features for n-table queries.

[0058] One or more embodiments described herein provide an ensemble architecture. For example, according to one or more embodiments, a graph-based approach provides multiple models to work together to enhance the accuracy of the use of the model.

[0059] Now turn to Figure 2, one or more embodiments described herein may utilize machine learning techniques to perform tasks, such as estimating the cardinality of a query using a query graph. More specifically, one or more embodiments described herein may combine and utilize rule-based decision making and artificial intelligence (AI) reasoning to complete the various operations described herein, i.e., estimating the cardinality of a query using a query graph. The phrase "machine learning" broadly describes the functionality of an electronic system that learns from data. A machine learning system, engine, or module may include a trainable machine learning algorithm that may be trained, such as in an external cloud environment, to learn a functional relationship between input and output, and the resulting model (sometimes referred to as a "trained neural network," "trained model," and / or "trained machine learning model") may be used, for example, to estimate the cardinality of a query using a query graph. In one or more embodiments, the machine learning functionality may be implemented using an artificial neural network (ANN) that has the ability to be trained to perform a function. In machine learning and cognitive science, ANN is a series of statistical learning models inspired by the biological neural networks of animals (particularly the brain). ANN can be used to estimate or approximate systems and functions that rely on a large number of inputs. Convolutional neural networks (CNNs) are a class of deep feedforward ANNs that are particularly useful in tasks such as, but not limited to, analyzing visual images and natural language processing (NLP). Recurrent neural networks (RNNs) are another class of deep feedforward ANNs and are particularly useful in tasks such as, but not limited to, unsegmented continuous handwriting recognition and speech recognition. Other types of neural networks are also known and can be used in accordance with one or more embodiments described herein.

[0060] ANN can be implemented as a so-called "neuromorphic" system of interconnected processor elements, which act as simulated "neurons" and exchange "messages" with each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that carry messages between biological neurons, the connections that carry electronic messages between simulated neurons in ANNs have digital weights corresponding to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience so that the ANN adapts to the input and can learn. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by pixels of an input image. After being weighted and transformed by a function determined by the network designer, the activation of these input neurons is then passed to other downstream neurons, which are generally referred to as "hidden" neurons. This process is repeated until the output neuron is activated. The activated output neuron determines which character is input. It should be understood that these same techniques can be applied to the case of estimating the cardinality of a query using a query graph as described herein.

[0061] Reference now Figure 2Describes in more detail the system for training and using machine learning models. In particular, Figure 2 A block diagram of components of a machine learning training and reasoning system 200 is depicted in accordance with one or more embodiments described herein. The machine learning training and reasoning system 200 performs training 202 and reasoning 204. During training 202, a training engine 216 trains a model (e.g., trained model 218) to perform a task, such as estimating the cardinality of a query using a query graph. Reasoning 204 is the process of implementing the trained model 218 in the context of a larger system (e.g., system 226) to perform a task, such as estimating the cardinality of a query using a query graph. Figure 2 All or part of the machine learning training and reasoning system 200 shown in FIG. 2 may be provided by, for example, Figure 1 It may be implemented in whole or a subset of computing environment 100.

[0062] Training 202 starts with training data 212, which can be structured or unstructured data. According to one or more embodiments described herein, training data 212 includes queries from a query workload. Training engine 216 receives training data 212 and model form 214. Model form 214 represents an untrained basic model. Model form 214 can have preset weights and biases, which can be adjusted during training. It should be understood that model form 214 can be selected from many different model forms depending on the task to be performed. For example, in the case where training 202 is to train a model to perform image classification, model form 214 can be a model form of CNN. Training 202 can be supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, etc., including combinations and / or multiple forms thereof. For example, supervised learning can be used to train a machine learning model to classify an object of interest in an image. To this end, training data 212 includes labeled images, which include images of objects of interest with associated tags (real situations) and other images that do not include objects of interest with associated tags. In this example, the training engine 216 takes as input a training image from the training data 212, makes a prediction for classifying the image, and compares the prediction to the known label. The training engine 216 then adjusts the weights and / or biases of the model based on the results of the comparison, such as by using back propagation. The training 202 can be performed multiple times (referred to as "epochs") until a suitable model is trained (e.g., the trained model 218).

[0063] Once trained, the trained model 218 can be used to perform inference 204 to perform tasks, such as estimating the cardinality of a query using a query graph. The inference engine 220 applies the trained model 218 to new data 222 (e.g., real-world, non-training data). For example, if the trained model 218 is trained to classify images of a particular object, such as a chair, the new data 222 may be images of chairs that are not part of the training data 212. In this way, the new data 222 represents data that has not yet been exposed to the trained model 218. The inference engine 220 makes a prediction 224 (e.g., a classification of an object in an image of the new data 222) and passes the prediction 224 to the system 226 (e.g., Figure 1 The computing environment 100 of the present invention. The system 226 can take actions, perform operations, perform analysis, etc. based on the prediction 224, including combinations and / or multiple forms thereof. In some embodiments, the system 226 can add and / or modify the new data 222 based on the prediction 224.

[0064] According to one or more embodiments, predictions 224 generated by inference engine 220 are periodically monitored and validated to ensure that inference engine 220 operates as expected. Based on the validation, additional training 202 can be conducted using trained model 218 as a starting point. Additional training 202 can include all or a subset of original training data 212 and / or new training data 212. According to one or more embodiments, training 202 includes updating trained model 218 to account for changes in expected input data.

[0065] Figure 3 A method 300 for estimating the cardinality of a query using a query graph according to one or more embodiments described herein is depicted. The method 300 may be performed by any suitable system, environment, and / or device, such as the computing environment 100, the machine learning training and reasoning system 200, etc., including combinations and / or multiples thereof. Now with reference at least to Figure 2 Method 300 is described in more detail, but is not limited thereto.

[0066] At block 302, a processing system (e.g., computing environment 100, machine learning training and reasoning system 200, etc., including combinations and / or multiple forms thereof) trains a machine learning model (e.g., trained model 218) using a query graph. To train the machine learning model, the processing system generates training queries from a query workload, or generates training queries based on relationships between tables in a database, and / or receives such training queries. Using queries from the training queries, the processing system constructs a query graph using input vectors of tables in the database anonymized using representations of characteristics of the database, and provides selectivity as a desired output (e.g., using representations of the queries as a graph). The processing system then constructs an initial join result distribution as a query graph set based on the training queries.

[0067] Training a machine learning model (e.g., trained model 218) is now described in more detail. Training data (e.g., training data 212) can be incrementally generated using collected query workloads. If the collected query workloads are not initially available, a synthetic workload for the database can be generated and used. The generated queries can be joint queries with random local predicates based on defined relationships in the database. If workload queries are collected, the workload queries can be simplified to the join portion of the query and the local predicates.

[0068] The training engine 216 takes as input a vector comprising the dimensions of each node in the query graph. According to an embodiment, the positioning of the nodes is based on geometric rules established in the graphical representation described herein. For the encoding of node features such as base table selectivity, connection sample selectivity, and high-frequency value selectivity, randomly collected samples can be scanned. Training does not require any non-random sampling techniques to maintain high-quality estimates. However, according to one or more embodiments described herein, if traces are available from previously run workload queries, selectivity can be obtained from these traces. For well-defined star patterns, samples of fact tables connected to full-dimensional tables produce improved results compared to independent samples. Accurate results can be obtained without samples of fact tables connected to full-dimensional tables at the expense of increased training time.

[0069] According to one or more embodiments described herein, any suitable amount of training data may be used to train the trained model 218. According to an embodiment, the amount of training data may depend on relevance to the workload. According to an embodiment, a training accuracy threshold and model size or training time may be implemented to guide the amount of training data required. Additionally and / or alternatively, runtime feedback of predicted estimates relative to actual cardinality while the workload is running may be used to continuously train or retrain a model as described herein.

[0070] Once training 202 has been performed (e.g., once training engine 216 has trained trained model 218), reasoning 204 can begin. In box 304, the processing system uses the trained machine learning model from box 302 to estimate the cardinality of the query. In other words, the processing system uses the trained machine learning from box 302 to perform reasoning. For example, for each enumerated connection, the optimizer calls the trained machine learning model to calculate the connection cardinality. The optimizer enumerates different permutations for the same set of N tables (or subsets). According to the example, the cardinality is the same, so the predictions are calculated once, cached, and reused as needed, resulting in improved functionality of the processing system. The trained machine learning model identifies the best fit among the query graph set and generates an associated cardinality estimate for the best fit, which is returned to the optimizer.

[0071] Reasoning 204 begins when a query is received by a reasoning engine 220 (also referred to as an "optimizer" or "database optimizer"). Once the database optimizer receives the query, the optimizer enumerates various permutations of join orders. For each of these join orders, the optimizer constructs an input vector that describes the join graph and one or more properties (which are further described herein). The reasoning engine 220 matches the input vector to patterns among available patterns of the query graph and applies a learning model to generate an associated prediction cardinality (e.g., prediction 224).

[0072] According to one or more embodiments described herein, the optimizer may enumerate different permutations of join orders for the same set of N tables (or subset), but the resulting cardinality for each join order is the same. In order to minimize any computational load (e.g., processing and / or memory) overhead of constructing input vectors and / or computing predictions using a model, the prediction results may be computed once for a set of N tables and cached for later reuse.

[0073] To compute intermediate cardinality with a subset of tables (e.g., for each join order enumerated), the inference engine 220 may apply the trained model 218 using the subset of tables and local predicates of the tables to generate improved cardinality for joins of these tables. For example, a query may have five tables, two of which are lookup joins (N:1). In this example, the trained model 218 only predicts the cardinality of joins for the other three tables, which are primarily filter joins. The cardinality does not change after joining the lookup tables. This reduces the size and complexity of the query graph, and also reduces the training queries required to train the machine learning model.

[0074] It should be understood that during inference 204, table and column distribution properties may not exactly match those in the queries used during training 202. If there are a large enough number of tables and training scenarios, there will be enough tables with a variety of distributions and predicates so that the trained model 218 will work even for new tables and columns that have not been seen exactly before.

[0075] At block 306, the processing system performs continuous training and / or completely retraining of the trained machine learning model from block 302. For example, new queries in a new query workload may be edited based on new workloads, new tables, changed data in tables, differences between estimates predicted by the trained model and actual cardinality from runtime feedback, etc. (including combinations and / or multiples thereof). A trained machine learning model may be retrained by adding or modifying a query graph set with a new query workload.

[0076] Since databases are usually dynamic, the information stored in the database changes over time. Therefore, if the underlying distribution of some tables changes significantly over time, a complete model retraining may be required (e.g., retraining the trained model 218 by performing training 202 again). A threshold can be implemented to indicate when the underlying distribution has changed significantly (e.g., enough to prove that retraining is correct). For example, the threshold can be the percentage difference between the frequency and / or histogram statistics collected for the table and the frequency and / or histogram statistics when the trained model 218 was fully trained for the last time. According to one or more embodiments described herein, retraining is performed based on the degree of change (e.g., the number of rows updated, deleted and / or inserted relative to the table size, including their combinations and / or multiple forms). According to one or more embodiments described herein, retraining is performed based on the number of tables created, deleted or changed in a way that affects the model (e.g., deleting columns, adding columns, etc., including their combinations and / or multiple forms).

[0077] The trained model 218 can also be continuously updated / trained. Through continuous training, the labels used for the query graph may become outdated and no longer reflect the current reality. Therefore, the queries previously used in training can be one or more of the following scenarios. First, queries that have not been executed recently in the training set. These queries still carry the correct labels. Second, recently executed queries. If these queries match any entry in the training set, they are added to the new query workload. Any hash function can be used to hash the queries in the training data 212 to discard redundant queries. Third, queries that are not in the training set and the model estimates are higher / lower than the actual cardinality by a certain amount (e.g., a threshold number, a threshold percentage difference, etc., including their combination and / or multiple forms) (e.g., an order of magnitude difference). These queries are added to the new query workload. Any other metric can also be used to trigger this scenario. According to one or more embodiments described herein, the next time the model is retrained, some or all of these queries are added as a set of new observed examples to retrain the trained model 218.

[0078] According to one or more embodiments described herein, a runtime feedback component showing the difference between predicted cardinality and actual cardinality can be used to indicate when a trained model 218 needs to have queries added for additional continuous training or added to the training workload for a complete retraining.

[0079] According to one or more embodiments, the trained machine learning model provides the ability to learn the distribution of connection results, provides induction that can handle unseen tables and columns, provides feature decomposition, enables ensemble architecture, etc., including combinations and / or multiple forms thereof. Additional processes may also be included. According to one or more embodiments described herein, method 300 may be applied to relational databases and / or other database technologies, such as graph databases, extensible markup language (XML) databases, etc., including combinations and / or multiple forms thereof. It should be understood that Figure 3 The processes depicted in the drawings are illustrative and other processes may be added or existing processes may be removed, modified or rearranged without departing from the scope of the present disclosure.

[0080] Figure 4 Query graphs 400, 410 are depicted in accordance with one or more embodiments described herein. The query graphs 400, 410 can be used during training 202 to train the trained model 218 and / or to perform reasoning 204 using the trained model 218. Training 202 and reasoning 204 are now described in more detail with reference to the query graphs 400, 410, but are not limited thereto.

[0081] According to one or more embodiments described herein, estimating the cardinality of a query is a regression problem in which the cardinality of a query graph (e.g., one or more of query graphs 400, 410) is predicted. The query graph may represent a complete query or a sub-portion of a query, such as different permutations of joins enumerated by a query optimizer. According to one or more embodiments described herein, any regression machine learning model may be used. The query graph is what the trained model 218 learns along with the true cardinality of the query. The target variable of the trained model 218 may be the selectivity of a cartesian join. Learning the selectivity instead of the actual cardinality enables the trained model 218 to have a normalized variable that can accelerate training 202 and increase the accuracy of the trained model 218. The selectivity may be transformed back to the cardinality of the output. The query graph may represent a sub-portion of a query plan. The target variable is the selectivity of a cartesian join.

[0082] In order for the trained model 218 to learn how to estimate the cardinality of a query, a valid list of features is defined to represent the properties of the query. A query graph (e.g., one or more of query graphs 400, 410) includes a graph, a vertex (or multiple vertices), an edge (or multiple edges), and the degree of the vertex.

[0083] Each query can be modeled as a graph G according to the following formula:

[0084] G={V,E,D}

[0085] V: is a finite set of vertices

[0086] is the set of edges.

[0087] Vertices are graph nodes that represent database relational tables. Adjacent vertices are tables connected by edges. Edges are undirected lines that represent binary connections between two tables. The degree of a vertex represents the number of tables connected to a given vertex.

[0088] The join operation has a binary nature (e.g., only two tables are joined at a time). Each join can be represented by two vertices connected by an undirected edge. For example, query graph 400 represents a first query. In this example, query graph 400 includes three vertices 401, 402, 403, which are connected by the edges shown. The three vertices 401-403 indicate that the query references three tables. Query graph 410 represents a second query and includes six vertices 411, 412, 413, 414, 415, 416 connected by the edges shown. The six vertices 411-406 indicate that the query references six tables. In the case where the query has only one table, a self-join node can be used to represent a graph with a single table.

[0089] According to one or more embodiments described herein, for any given query graph G, there are at least |T|-1 edges, where |T| represents the number of tables. The query space is restricted between +1 and -1 values. Restricting the space helps to train the trained model 218 with a normalized range of values ​​that can be adjusted to accommodate a large number of tables in each query. Now describe how to place nodes in such a graph space.

[0090] The generation of a query graph (e.g., one or more of query graphs 400, 410) is based on the number of tables provided in the query and join columns. The coordinates of each node can represent some features describing the base table. The attributes of the edge can be used to describe various aspects of the connection. For example, the line weight and the length of the edge can describe the join column of each binary node, and other attributes such as the type of connection can be represented. For example, the direction of the edge can represent the type of connection, where no direction indicates an inner join, a single direction represents a left outer join or a right outer join and the arrow points to the side that produces a null value (e.g., T1→T2 describes T1LOJ T2), and a bidirectional edge represents a full outer join (e.g., T1←→T2 describes T1FOJ T2). According to one or more embodiments described herein, the characteristics of each node can be the base table selectivity after applying a local predicate, the total selectivity of high-frequency values, and / or any other characteristics that demonstrate the effect of applying a local predicate on the base table cardinality.

[0091] Building a query graph (eg, one or more of query graphs 400, 410) is now described. To build a query graph, nodes and edges are created, an initial position for each node is determined, and then the node positions are refined / optimized.

[0092] Reference now Figure 5 Describes the creation of nodes and edges, Figure 5 A spatial graph 500 showing two tables with their connection columns is depicted according to one or more embodiments described herein. Each set of connection columns between any two nodes is considered as an entity of the connection feature. For example, if there is a connection between a first table T1 and a second table T2 on the following columns: T1.c1=T2.c1 and T1.c5=T2.c5, then a node in the graph is created such as Figure 5 As shown. The edge between the first table T1 and the second table T2 represents the connection column. According to the example, the line weight or style of the edge (e.g., a dotted line) can represent the operator between the connection columns: inner join, outer join, etc., where different line weights or styles represent different types of connections.

[0093] Reference now Figure 6 Describes how to determine the initial position of each node. Figure 6 Depicts each node representing the initial coordinates in the graph space (from Figure 5) features. Each node has at least two dimensions that represent the coordinates of the node in the graph. The base table cardinality together with the total selectivity of the high-frequency values ​​is effectively used as the initial position of each node in the graph. For example, Figure 6 Shown for the Figure 5 The two-dimensional initial position of each node of the first table T1 and the second table T2. The initial positions of the nodes of the first table T1 are 0.33 and 0.01...0.001. The initial positions of the nodes of the second table T2 are 0.51 and 0.03...0.01. The first value (0.33 for T1 and 0.51 for T2) represents the selectivity of the base table cardinality after applying the local predicate. The second value (0.01...0.001 for T1 and 0.03...0.01 for T2) represents the total selectivity of x high-frequency values. The second value is used to determine whether the application of the local predicate has changed the selectivity of any highly skewed values ​​(if any). According to one or more embodiments described herein, the total selectivity of the first x values ​​is calculated from a randomly collected sample. According to one or more embodiments described herein, the total selectivity of the first x values ​​is calculated from some predefined materialized samples or tables from which statistics were previously collected. According to an example, for each table, the sample size can be between 2000 and 5000 rows. In some cases, such as for smaller tables, sampling can be skipped.

[0094] Reference now Figure 7 Describe the optimized node location, Figure 7Depicted is a result query graph 700 with optimized node positioning for a three-table query according to one or more embodiments described herein. Relying on a query graph to represent a query and then using underlying distribution properties introduces many challenges. First, ensure that the graph has the same representation given the same situation (e.g., the same table, the same connection, and the same local predicate, while the actual distribution does not change). Second, ensure that nodes do not overlap in position. Third, generalize the query graph to avoid encoding table names, predicates, and connections to be able to handle unseen tables, change tables with unseen predicates and data in queries, etc., including combinations and / or multiple forms thereof. To address these challenges, one or more embodiments described herein provide for optimizing the positioning of nodes in a query graph given the initial positions described herein. One possible method for optimizing the positioning of nodes in a query graph is to use the optimization algorithm described in "Graph drawing by force-directed placement" (Software: Practice and experience 21.11 (1991): 1129-1164, Fruchterman, Thomas MJ, and Edward M. Reingold). This method computes the attractive forces between adjacent vertices in the query graph as well as the repulsive forces of the vertices. Figure 7 The result query graph 700 using the optimized three-table query is shown. According to one or more embodiments described herein, additional dimensions representing join selectivity can be added to the query graph, or features can be included in the node coordinates generated by the optimization algorithm. Expressions on columns can also be encoded and added to attributes.

[0095] According to one or more embodiments described herein, additional dimensions may be considered to further optimize the positioning of nodes: the ratio of table sizes without any predicates, the ratio of distinct values, and the selectivity of local predicates on the actual connection sample.

[0096] In order to train the trained model 218 to generalize on unseen tables, the ratio of the table size of the child node to the parent node can be added. This ratio provides an indication of how large the table was initially before the join.

[0097] The ratio of distinct values ​​provides the underlying distribution to determine the join. The distinct values ​​between nodes provide the ability to determine if the relationship is a primary-foreign key or if the join is between two primary keys.

[0098] The selectivity of the join sample determines the impact of the predicate on the actual join sample. For example, for each unique join, the selectivity of the predicate can be measured on a sample of 2000 rows collected. The sample is not joined, which may lead to the 0-tuple problem on the actual join of the node.

[0099] Figure 8 Depicted is a result query graph 800 with updated locations according to one or more embodiments described herein. In this example, the result query graph 800 is Figure 7 Result query graph 800 is an updated version of result query graph 700. Result query graph 800 includes an updated position of the y-axis with the selectivity of the predicates on the connection samples. In particular, result query graph 700 is generated after using the optimization algorithm as described herein, and result query graph 800 displays the updated position of the nodes using the selectivity of the connection samples.

[0100] Fig. 9A , 9B 9C together depict an example of constructing a query graph for a given query according to one or more embodiments described herein. In this example, the input to the trained model 218 is the features of each query. Examples of features include the base table selectivity of each node, the coordinates (x, y) of each node, which are generated using the techniques described herein. Fig. 9A Depicted is an example query 900 in accordance with one or more embodiments described herein. The sample query 900 uses, for example, IMDb. Fig. 9B Depicts each node representing the initial coordinates in the graph space (from Fig. 9C ). In this example, base table attributes (BTA) include base table selectivity (e.g., considering local predicates) and selectivity of high frequency values ​​(e.g., representing data distribution). The processing system can normalize the coordinates to summarize the graph, such as using the Fruchterman and Reingold algorithm as described herein. Join features utilize ratios of table sizes, ratios of distinct values, and / or selectivity of predicates on join samples, as described herein. Fig. 9C Depicted is a query graph representation 902 of a sample query 900 run on the Internet Movie Database (IMDb).

[0101] Fig.10 A flowchart of a method for training a machine learning model to estimate the cardinality of a query using a query graph according to one or more embodiments described herein is depicted. The method 1000 may be performed by any suitable system, environment, and / or device, such as the computing environment 100, the machine learning training and reasoning system 200, etc., including combinations and / or multiple forms thereof. Now at least reference is made to Figure 2 Method 1000 is described in more detail, but is not limited thereto.

[0102] At box 1002, a processing system generates training queries from a query workload, or generates training queries based on relationships between tables in a database (e.g., referential integrity constraints may be predefined or discovered), and / or receives such training queries. At box 1004, using queries from the training queries, the processing system constructs a query graph using input vectors of tables in the database anonymized by representations using characteristics of the database. For example, constructing a query graph representation for each training query in the training queries includes: using input vectors of tables anonymized by representations using characteristics; and providing selectivity as a desired output (e.g., the location of nodes may be based on geometric rules established in the query graph representation). At box 1006, the processing system provides (e.g., calculates) selectivity as a desired output (e.g., using a representation of the query as a graph). At box 1008, the processing system constructs an initial connection result distribution as a query graph set based on the training queries.

[0103] Flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent the module, segment or part of instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function noted in the frame may not occur in the order noted in the figure.For example, two frames shown continuously can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of the frames in each frame of the block diagram and / or flow chart illustration and the block diagram and / or flow chart illustration can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special hardware and computer instructions.

[0104] The description of various embodiments of the present invention has been given for the purpose of illustration, but it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications or technical improvements over existing technologies on the market, or to enable other persons of ordinary skill in the art to understand the embodiments described herein.

Claims

1. A computer-implemented method comprising: To train a machine learning model, follow these steps: generating a training query set using at least one of a query workload and a relationship between tables in a database; For each training query in the training query set, construct a query graph; For each training query in the training query set, calculating a selectivity based at least in part on the query graph; as well as Based at least in part on the training query set, an initial join result distribution is constructed as a query graph set.

2. The computer-implemented method of claim 1 , wherein: Constructing the query graph for each of the training queries includes: using an input vector of anonymizing a table by using a representation of a feature; and providing the selectivity as a desired output.

3. The computer-implemented method of claim 1 , further comprising: After training the machine learning model, inference is performed using the machine learning model.

4. The computer-implemented method of claim 3, wherein: Performing the inference includes: For each combination of joins in the query, construct a query graph; and A best fit among the set of query graphs is identified to generate an associated cardinality estimate.

5. The computer-implemented method of claim 4, wherein: Calculating the connection cardinality includes: enumerating different arrangements for the same group of N tables.

6. The computer-implemented method of claim 5, wherein: Calculating the join cardinality includes: for each enumerated join order, calculating an intermediate cardinality using a subset of tables, wherein the machine learning model is called using the subset of tables and a local predicate.

7. The computer-implemented method of claim 1 , further comprising: Continuous training is performed on the machine learning model.

8. The computer-implemented method of claim 7, wherein: Performing the continuous training on the machine learning model includes: Based on the workload, the new table, and the changes to the data, editing a new query in the new query workload using input including at least one change selected from the group comprising the new query; The query graph collection is updated.

9. The computer-implemented method of claim 1 , further comprising: Based on the workload, the new table, and the changes to the data, editing a new query in the new query workload using input including at least one change selected from the group comprising the new query; as well as Retraining the machine learning model is performed using the new query workload to create updates, and the updates are used to update the query graph set.

10. A system comprising: a memory including computer readable instructions; as well as A processing device configured to execute the computer-readable instructions, wherein the computer-readable instructions control the processing device to perform operations, the operations comprising: To train a machine learning model, follow these steps: generating a training query set using at least one of a query workload and a relationship between tables in a database; For each training query in the training query set, construct a query graph; For each training query in the training query set, computing a selectivity based at least in part on the query graph; and Based at least in part on the training query set, an initial join result distribution is constructed as a query graph set.

11. The system according to claim 10, wherein: Constructing the query graph for each of the training queries includes: using an input vector of anonymizing a table by using a representation of a feature; and providing the selectivity as a desired output.

12. The system according to claim 10, wherein: The operation also includes: after training the machine learning model, performing inference using the machine learning model.

13. The system according to claim 12, wherein: Performing the inference includes: constructing a query graph for each combination of joins in the query; and A best fit among the set of query graphs is identified to generate an associated cardinality estimate.

14. The system according to claim 13, wherein: Calculating the connection cardinality includes: enumerating different arrangements for the same group of N tables.

15. The system of claim 14, wherein: Calculating the join cardinality includes: for each enumerated join order, calculating an intermediate cardinality using a subset of tables, wherein the machine learning model is called using the subset of tables and a local predicate.

16. The system of claim 10, wherein: The operation also includes: performing continuous training on the machine learning model.

17. The system of claim 16, wherein: Performing the continuous training on the machine learning model includes: Based on the workload, the new table, and the changes to the data, editing a new query in the new query workload using input including at least one change selected from the group comprising the new query; The query graph collection is updated.

18. The system of claim 10, wherein: The operations also include: Based on the workload, the new table, and the changes to the data, editing a new query in the new query workload using input including at least one change selected from the group comprising the new query; and Retraining the machine learning model is performed using the new query workload to create updates, and the updates are used to update the query graph set.

19. A computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions being executable by a processor to cause the processor to perform operations comprising: To train a machine learning model, follow these steps: generating a training query set using at least one of a query workload and a relationship between tables in a database; For each training query in the training query set, construct a query graph; For each training query in the training query set, calculating a selectivity based at least in part on the query graph; as well as Based at least in part on the training query set, an initial join result distribution is constructed as a query graph set.

20. The computer program product of claim 19, wherein: Constructing the query graph for each of the training queries includes: using an input vector of anonymizing a table by using a representation of a feature; and providing the selectivity as a desired output.

21. The computer program product of claim 19, wherein: The operation also includes: after training the machine learning model, performing inference using the machine learning model.

22. The computer program product of claim 21, wherein: Performing the inference includes: constructing a query graph for each combination of joins in the query; and A best fit among the set of query graphs is identified to generate an associated cardinality estimate.

23. The computer program product of claim 22, wherein: Calculating the join cardinality includes: enumerating different arrangements for the same set of N tables, wherein calculating the join cardinality includes: for each enumerated join order, using a subset of the tables to calculate the intermediate cardinality, wherein the machine learning model is called using the subset of the tables and local predicates.

24. A computer-implemented method comprising: To train a machine learning model, follow these steps: generating a training query set using at least one of a query workload and a relationship between tables in a database; For each training query in the training query set, construct a query graph; For each training query in the training query set, calculating a selectivity based at least in part on the query graph; as well as constructing an initial connection result distribution as a query graph set based at least in part on the training query set; After training the machine learning model, perform inference using the machine learning model by following the steps below: Calculate the join cardinality for each join statement in the query; and identifying a best fit among the set of query graphs to generate an associated cardinality estimate; and Continuous training of the machine learning model is performed through the following steps: Based on the workload, the new table, and the changes to the data, editing a new query in the new query workload using input including at least one change selected from the group comprising the new query; and The query graph collection is updated.

25. A computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions being executable by a processor to cause the processor to perform operations comprising: To train a machine learning model, follow these steps: generating a training query set using at least one of a query workload and a relationship between tables in a database; For each training query in the training query set, construct a query graph; For each training query in the training query set, calculating a selectivity based at least in part on the query graph; as well as constructing an initial connection result distribution as a query graph set based at least in part on the training query set; After training the machine learning model, perform inference using the machine learning model by following the steps below: Calculate the join cardinality for each join statement in the query; and identifying a best fit among the set of query graphs to generate an associated cardinality estimate; and Continuous training of the machine learning model is performed through the following steps: Based on the workload, the new table, and the changes to the data, editing a new query in the new query workload using input including at least one change selected from the group comprising the new query; and The query graph collection is updated.

Images