Data query method and apparatus, and medium and electronic device

Abstract

The present disclosure relates to a data query method and apparatus, and a medium and an electronic device. The method comprises: in response to a structured query statement being received, determining, from among reference cardinal-number estimation models which are loaded by a database engine and respectively correspond to storage units, a target cardinal-number estimation model which corresponds to the structured query statement, wherein the reference cardinal-number estimation models are obtained by means of training outside the database engine; and estimating the cardinal number of the structured query statement by using the target cardinal-number estimation model.

Description

Data query method, device, medium and electronic equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority to an application filed in China with application number 2023109231462, filed on July 25, 2023. The disclosure of the application is hereby incorporated as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of database technology, and in particular to a data query method, device, medium, and electronic device. Background Art

[0004] The query optimizer is one of the most important components in databases and big data systems.

[0005] Summary of the Invention

[0006] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] In a first aspect, the present disclosure provides a data query method, applied to a database engine, comprising:

[0008] In response to receiving a structured query statement, determining a target cardinality estimation model corresponding to the structured query statement from reference cardinality estimation models corresponding to respective storage units loaded by the database engine, wherein the reference cardinality estimation models are trained outside the database engine;

[0009] The target cardinality estimation model is used to estimate the cardinality of the structured query statement.

[0010] In a second aspect, the present disclosure provides a data query device, applied to a database engine, comprising:

[0011] a determination module configured to, in response to receiving a structured query statement, determine a target cardinality estimation model corresponding to the structured query statement from reference cardinality estimation models corresponding to the respective storage units loaded by the database engine, wherein the reference cardinality estimation models are trained outside the database engine;

[0012] An estimation module is used to estimate the cardinality of the structured query statement using the target cardinality estimation model.

[0013] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the data query method provided in the first aspect of the present disclosure.

[0014] In a fourth aspect, the present disclosure provides an electronic device, comprising:

[0015] a storage device having a computer program stored thereon;

[0016] A processing device is used to execute the computer program in the storage device to implement the steps of the data query method provided in the first aspect of the present disclosure.

[0017] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings:

[0019] Fig. 1 is a flow chart showing a data query method according to an exemplary embodiment.

[0020] FIG2 is a schematic diagram showing a big data system integration framework for a cardinality estimation model according to an exemplary embodiment.

[0021] Fig. 3 is a block diagram showing a data query device according to an exemplary embodiment.

[0022] Fig. 4 is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0023] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0025] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0026] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0030] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0031] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0034] In the technical solution disclosed in the present invention, the reference cardinality estimation model corresponding to each storage unit of the database or big data system is trained outside the database engine and loaded into the database engine, that is, the reference cardinality estimation model training is implemented outside the database engine, and the reference cardinality estimation model inference is implemented within the database engine. In this way, a big data system integration framework for the cardinality estimation model can be provided. For cardinality estimation with high real-time requirements, the database engine that can quickly process transactions is directly implemented based on the reference cardinality estimation model loaded thereon, which can greatly improve the rate of cardinality estimation and meet the user's low latency requirements for data query. In addition, training the reference cardinality estimation model outside the database engine makes it easy to update or even replace the reference cardinality estimation model without modifying the database engine, thereby improving the scalability of the framework. For Structured Query Language (SQL) queries input by users, databases and big data systems often rely on query optimizers to generate high-quality execution plans for the execution engine, thereby achieving efficient query execution time, allowing users to obtain correct results while having a better user experience. Cardinality estimation (CE) plays a crucial role in the query optimizer. For each query, it estimates the result set size for each subquery plan, guiding the query optimizer in developing a more optimal execution plan. For a complex SQL query, the query optimizer often needs to call the cardinality estimation procedure hundreds or even thousands of times. Therefore, improving the speed of a single cardinality estimation is crucial for meeting users' low-latency query requirements.

[0035] Figure 1 is a flowchart of a data query method according to an exemplary embodiment, wherein the method can be applied to a database engine. The database engine, also known as the database kernel, is a core functional set that completes data operations of a database management system (DBMS). The components outside the database engine are other components in the database server other than the database engine. The database engine generally includes a parser, a query optimizer, and an execution engine. The parser is used to parse the structured query statements input by the user in terms of lexical and grammatical terms to determine the user's query intent; the query optimizer is used to generate a corresponding execution plan based on the parsed results of the statement; and the execution engine is used to perform specific execution according to each operation step in the plan in accordance with the execution plan to obtain the query results required by the user. In the query optimizer, cardinality estimation (CE) plays a very important role. For each query, its function is to estimate the size of the result set for each subquery plan of the query, thereby guiding the query optimizer to plan a better execution plan.

[0036] As shown in FIG1 , the data query method may include the following S101 and S102 .

[0037] In S101, in response to receiving a structured query statement, a target cardinality estimation model corresponding to the structured query statement is determined from the reference cardinality estimation models corresponding to each storage unit loaded by the database engine, and the reference cardinality estimation model is trained outside the database engine.

[0038] In the present disclosure, the structured query statement carries information such as table identifiers (eg, table names) to be queried, connection conditions between tables to be queried, and output restriction conditions of the structured query statement.

[0039] In actual application scenarios, when users perform data queries, they may need to query multiple tables, and the connection methods between the tables may be the same or different. For example, the connection method between the tables in a structured query statement may be a mixed connection of outer joins and inner joins, not just a single connection method.

[0040] The output restriction conditions carried in the structured query statement are restrictions on the query output results. There are two types: one is to limit the number of output results, and the other is to limit the starting position and offset of the data.

[0041] Specifically, when data query is needed, the client constructs a structured query statement (ie, SQL statement) according to actual needs and sends the structured query statement to the database engine. After receiving the structured query statement, the database engine starts the data query operation.

[0042] A corresponding reference cardinality estimation model is constructed for each storage unit in a database or big data system, where a storage unit can be a database table or a partition of a database table. After the cardinality estimation model corresponding to each storage unit is trained outside the database engine, it can be stored in a preset storage location outside the database engine. The database engine can then load the cardinality estimation model corresponding to each storage unit from the preset storage location.

[0043] The reference cardinality estimation model corresponding to each storage unit of the database or big data system is trained outside the database engine and loaded into the database engine. That is, the reference cardinality estimation model training is implemented outside the database engine, while the reference cardinality estimation model inference is implemented within the database engine, as shown in Figure 2.

[0044] In S102 , the cardinality of the structured query statement is estimated using a target cardinality estimation model.

[0045] In the above technical solution, the reference cardinality estimation model corresponding to each storage unit of the database or big data system is trained outside the database engine and loaded into the database engine, that is, the reference cardinality estimation model training is implemented outside the database engine, and the reference cardinality estimation model inference is implemented within the database engine. In this way, a big data system integration framework for cardinality estimation models can be provided. For cardinality estimation with high real-time requirements, a database engine that can quickly process transactions is directly implemented based on the reference cardinality estimation model loaded thereon, which can greatly improve the rate of cardinality estimation and meet the user's low-latency requirements for data queries. In addition, training the reference cardinality estimation model outside the database engine makes it easier to update or even replace the reference cardinality estimation model without modifying the database engine, thereby improving the scalability of the framework.

[0046] The reference cardinality estimation model can be, for example, a machine learning model such as an autoregressive model or a tree-based model. The autoregressive model uses an autoregressive network as its network structure, while the tree-based model can use a sum-product network or a Bayesian network as its network structure. Machine learning models can more accurately characterize the distribution of multidimensional data. Therefore, cardinality estimation models based on machine learning can obtain more accurate estimation results, i.e., high prediction accuracy.

[0047] Preferably, the above-mentioned reference cardinality estimation model can adopt a lightweight tree model. The model parameter scale of the lightweight model is small. In this way, a cardinality estimation model with higher prediction accuracy can be trained through a small number of training samples. In addition, it can reduce the load pressure of the database engine and increase the rate of cardinality estimation, thereby improving data query efficiency.

[0048] The following describes in detail the specific implementation method for determining the target cardinality estimation model corresponding to the structured query statement from the reference cardinality estimation models corresponding to each storage unit loaded from the database engine in S101. Specifically, this can be achieved through the following steps (1) and (2):

[0049] Step (1): Determine the table to be queried corresponding to the structured query statement.

[0050] In the present disclosure, since the structured query statement carries table identifiers (eg, table names) to be queried, the tables corresponding to the table identifiers can be used as tables to be queried.

[0051] Step (2): Determine the reference cardinality estimation model corresponding to the target storage unit among all reference cardinality estimation models as the target cardinality estimation model corresponding to the structured query statement.

[0052] In the present disclosure, the target storage unit is the storage unit included in the table to be queried. When the storage unit is a data table, the target storage unit is the table to be queried; when the storage unit is a partition, the target storage unit includes each partition included in the table to be queried.

[0053] In addition, before training the reference cardinality estimation model outside the database engine, it is necessary to obtain training samples, that is, to perform training sample sampling. Among them, training sample sampling can be performed in a variety of ways. In one embodiment, the database engine performs training sample sampling, that is, the database engine samples the data table in the database. At this time, the training samples of the reference cardinality estimation model come from the database. In this way, although the integration of kernel functions can be achieved, in the database engine, due to the existence of the partition key, the data of each partition often has its own inclination (that is, the data distribution is uneven), resulting in the inability to achieve unbiased sampling (meaning that the category ratio of the real sample population is maintained in the training set), thereby affecting the quality of the training samples and the inability to guarantee the prediction accuracy of the cardinality estimation model.

[0054] In view of this, in another embodiment, sample sampling is performed outside the database engine, that is, data sampling is performed outside the database engine from the upstream data source of the database engine (for example, Hadoop Distributed File System (HDFS), Kafaka, etc.), that is, the training samples of the reference cardinality estimation model come from the upstream data source of the database engine.

[0055] As shown in FIG2 , a training service is built outside the database engine to build a high-quality cardinality estimation model for a database table or partition in a database or big data system. The training service obtains training samples from an upstream data source, including small sample A11, small sample A12, small sample A21, small sample A22, ..., small sample An1, small sample An2, and trains models B11, B12, B21, B22, ..., Bn1, and Bn2 corresponding to partitions 11, 12, 21, 22, ..., n1, and n2, respectively, based on these training samples. The database in FIG2 can be, for example, ClickHouse, which is a columnar storage database with an MPP architecture for online analytical processing (OLAP) that was open-sourced by Russia's Yandex in 2016. It can generate analytical data reports in real time using SQL queries. The full name of ClickHouse is Click Stream, Data WareHouse.

[0056] Current databases or big data systems already have built-in import services. These services clean upstream data sources and perform other data preprocessing before importing them into the system. These services also maintain complete information about upstream data sources. The external database engine sampling process described in this disclosure interacts with this import service to obtain all the information needed to sample data.

[0057] When the storage unit is a database table, the upstream data source is the original data and is not affected by the partition key, thus achieving global unbiased sampling. When the storage unit is a partition of a database table, sampling within the database engine can be avoided and the upstream data source can be repartitioned, thus achieving unbiased sampling at the partition level. This allows for the acquisition of high-quality training samples, thereby improving the prediction accuracy of the cardinality estimation model. In addition, both model training and sample sampling are performed outside the database engine, achieving seamless integration of the sampling and training processes, forming a complete model training pipeline.

[0058] When the query optimizer estimates the result set size for each subquery plan, it can invoke the inference process of the cardinality estimation model. This process requires extremely high real-time performance, typically in milliseconds. To this end, the reference cardinality estimation model in this disclosure can be described in the same compiled programming language as the database engine (e.g., CPP, Java, Rust, etc.), that is, the cardinality estimation model inference is implemented using the same compiled programming language as the database engine.

[0059] Specifically, after training the reference cardinality estimation model for each storage unit outside the database engine, it can be converted into a format that can be parsed by the programming language developed by the database engine. This allows the database engine to directly use the loaded cardinality estimation model for cardinality estimation without further format conversion, thereby accelerating the inference process and improving the cardinality estimation rate.

[0060] Furthermore, tree models typically only support conditional inference on discrete variables (e.g., A=3), while structured query statements in the database field often need to support conditional inference on continuous variables (e.g., A>3). Therefore, the present disclosure can further accelerate the entire inference process by converting conditional inference on continuous variables into conditional inference on discrete variables and utilizing the common matrix operation library of the database engine's compiled programming language, thereby improving the rate of cardinality estimation.

[0061] Since the data in the database is updated dynamically, new database tables or partitions may be added, and the data distribution in the database tables or partitions may also change. Therefore, the reference cardinality estimation model corresponding to each database table or partition can be regularly updated and trained outside the database engine. In other words, the reference cardinality estimation model is regularly updated and trained outside the database engine. In this case, the above data query method can also include the following steps:

[0062] The database engine periodically loads the reference cardinality estimation model from outside the database engine.

[0063] In the present disclosure, during the normal operation stage of the database or big data system, the reference cardinality estimation model corresponding to each storage unit can be regularly loaded from a preset storage location outside the database engine through the first thread in the database engine, that is, each model in the database engine is updated to adapt to the changes in data distribution brought about by time and data updates, thereby ensuring the accuracy of the cardinality estimation and meeting the query requirements of the current structured query statement.

[0064] Because database data is dynamically updated, new tables or partitions may be added, and the data distribution within them may change. Therefore, the database engine can periodically retrieve and store the total number of rows in each storage unit. This is a common parameter used in cardinality estimation. Regularly retrieving and storing the total number of rows in each storage unit not only accelerates cardinality estimation but also ensures its accuracy.

[0065] In the present disclosure, during the normal operation of the database or big data system, the total number of rows of each storage unit can be regularly obtained and stored by the second thread in the database engine, that is, the total number of rows of each storage unit can be updated.

[0066] The loading frequency of the first thread and the updating frequency of the second thread can be determined by the updating frequency of the database storage unit. Since both the first thread and the second thread are lightweight threads, their impact on the runtime load of the database engine can be ignored.

[0067] FIG3 is a block diagram of a data query device according to an exemplary embodiment, wherein the device 300 can be applied to a database engine. As shown in FIG3 , the device 300 includes:

[0068] Determining module 301 is configured to, in response to receiving a structured query statement, determine a target cardinality estimation model corresponding to the structured query statement from reference cardinality estimation models corresponding to respective storage units loaded by the database engine, wherein the reference cardinality estimation models are trained outside the database engine;

[0069] The estimation module 302 is configured to estimate the cardinality of the structured query statement using the target cardinality estimation model.

[0070] In the above technical solution, the reference cardinality estimation model corresponding to each storage unit of the database or big data system is trained outside the database engine and loaded into the database engine, that is, the reference cardinality estimation model training is implemented outside the database engine, and the reference cardinality estimation model inference is implemented within the database engine. In this way, a big data system integration framework for cardinality estimation models can be provided. For cardinality estimation with high real-time requirements, a database engine that can quickly process transactions is directly implemented based on the reference cardinality estimation model loaded thereon, which can greatly improve the rate of cardinality estimation and meet the user's low-latency requirements for data queries. In addition, training the reference cardinality estimation model outside the database engine makes it easier to update or even replace the reference cardinality estimation model without modifying the database engine, thereby improving the scalability of the framework.

[0071] Optionally, the training samples of the reference cardinality estimation model come from an upstream data source of the database engine.

[0072] Optionally, the reference cardinality estimation model is regularly updated and trained outside the database engine;

[0073] The apparatus 300 further includes:

[0074] A loading module is used for the database engine to periodically load the reference cardinality estimation model from outside the database engine.

[0075] Optionally, the database engine periodically obtains and stores the total number of rows of each storage unit.

[0076] Optionally, the reference cardinality estimation model is described using the same compiled programming language as that of the database engine.

[0077] Optionally, the reference cardinality estimation model is a tree model.

[0078] Optionally, the determining module 301 includes:

[0079] A first determining submodule is used to determine a table to be queried corresponding to the structured query statement;

[0080] The second determination submodule is used to determine the reference cardinality estimation model corresponding to the target storage unit in all the reference cardinality estimation models as the target cardinality estimation model corresponding to the structured query statement, wherein the target storage unit is the storage unit contained in the table to be queried.

[0081] The present disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the above-mentioned data query method provided by the present disclosure.

[0082] Reference is now made to FIG4 , which illustrates a schematic diagram of the structure of an electronic device (e.g., a terminal device or server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG4 is merely an example and should not limit the functionality or scope of use of the embodiments of the present disclosure.

[0083] As shown in Figure 4, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0084] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 shows the electronic device 600 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0085] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0086] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0087] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0088] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0089] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to receiving a structured query statement, determines a target cardinality estimation model corresponding to the structured query statement from the reference cardinality estimation models loaded by the database engine and corresponding to each storage unit, wherein the reference cardinality estimation model is trained outside the database engine; and estimates the cardinality of the structured query statement using the target cardinality estimation model.

[0090] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The modules described in the embodiments of the present disclosure may be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the module itself. For example, an estimation module may also be described as a "module for estimating the cardinality of the structured query statement using the target cardinality estimation model."

[0093] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0094] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] According to one or more embodiments of the present disclosure, Example 1 provides a data query method, applied to a database engine, comprising:

[0096] In response to receiving a structured query statement, determining a target cardinality estimation model corresponding to the structured query statement from reference cardinality estimation models corresponding to respective storage units loaded by the database engine, wherein the reference cardinality estimation models are trained outside the database engine;

[0097] The target cardinality estimation model is used to estimate the cardinality of the structured query statement.

[0098] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the training samples of the reference cardinality estimation model come from an upstream data source of the database engine.

[0099] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein the reference cardinality estimation model is regularly updated and trained outside the database engine;

[0100] The method further comprises:

[0101] The database engine periodically loads the reference cardinality estimation model from outside the database engine.

[0102] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, wherein the database engine periodically obtains and stores the total number of rows of each storage unit.

[0103] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, wherein the reference cardinality estimation model is described using the same compiled programming language as the database engine.

[0104] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein the reference cardinality estimation model is a tree model.

[0105] According to one or more embodiments of the present disclosure, Example 7 provides the method of any one of Examples 1 to 6, wherein determining a target cardinality estimation model corresponding to the structured query statement from the reference cardinality estimation models loaded by the database engine and corresponding to the respective storage units includes:

[0106] Determining a table to be queried corresponding to the structured query statement;

[0107] The reference cardinality estimation model corresponding to the target storage unit among all the reference cardinality estimation models is determined as the target cardinality estimation model corresponding to the structured query statement, wherein the target storage unit is the storage unit included in the table to be queried.

[0108] According to one or more embodiments of the present disclosure, Example 8 provides a data query device, applied to a database engine, comprising:

[0109] a determination module configured to, in response to receiving a structured query statement, determine a target cardinality estimation model corresponding to the structured query statement from reference cardinality estimation models corresponding to the respective storage units loaded by the database engine, wherein the reference cardinality estimation models are trained outside the database engine;

[0110] An estimation module is used to estimate the cardinality of the structured query statement using the target cardinality estimation model.

[0111] According to one or more embodiments of the present disclosure, Example 9 provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the data query method described in any one of Examples 1-7.

[0112] According to one or more embodiments of the present disclosure, Example 10 provides an electronic device, including:

[0113] a storage device having a computer program stored thereon;

[0114] A processing device is used to execute the computer program in the storage device to implement the steps of the data query method described in any one of Examples 1-7.

[0115] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0116] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0117] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A data query method, applied to a database engine, comprising: In response to receiving a structured query statement, determining a target cardinality estimation model corresponding to the structured query statement from reference cardinality estimation models loaded by the database engine and corresponding to each storage unit, wherein the reference cardinality estimation model is trained outside the database engine; The target cardinality estimation model is used to estimate the cardinality of the structured query statement. 2 . The method according to claim 1 , wherein the training samples of the reference cardinality estimation model come from an upstream data source of the database engine.

3. The method according to claim 1 or 2, wherein the reference cardinality estimation model is regularly updated and trained outside the database engine; The method further comprises: The database engine periodically loads the reference cardinality estimation model from outside the database engine.

4. The method according to any one of claims 1 to 3, wherein the database engine periodically obtains and stores the total number of rows of each storage unit.

5. The method according to any one of claims 1-4, wherein the reference cardinality estimation model is described in the same compiled programming language as the database engine.

6. The method according to any one of claims 1-5, wherein the reference cardinality estimation model is a tree model.

7. The method according to any one of claims 1 to 6, wherein determining a target cardinality estimation model corresponding to the structured query statement from the reference cardinality estimation models loaded by the database engine and corresponding to each storage unit respectively comprises: Determine a table to be queried corresponding to the structured query statement; The reference cardinality estimation model corresponding to the target storage unit among all the reference cardinality estimation models is determined as the target cardinality estimation model corresponding to the structured query statement, wherein the target storage unit is the data to be queried. The storage unit contained in the table.

8. A data query device, applied to a database engine, comprising: a determination module, configured to, in response to receiving a structured query statement, determine a target cardinality estimation model corresponding to the structured query statement from reference cardinality estimation models loaded by the database engine and corresponding to each storage unit, wherein the reference cardinality estimation model is trained outside the database engine; An estimation module is used to estimate the cardinality of the structured query statement using the target cardinality estimation model.

9. A computer readable medium having a computer program stored thereon, wherein when the program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

12. A computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images