Data query method and device for distributed database
By extracting keyword features and generating keyword density features from query statements in a distributed database, and using a cardinality estimation model to estimate the data cardinality, the problem of insufficient data query efficiency and accuracy in distributed databases is solved, and more efficient and accurate estimation of the target data quantity is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING OCEANBASE TECHNOLOGY CO LTD
- Filing Date
- 2023-12-18
- Publication Date
- 2026-05-29
AI Technical Summary
Existing distributed databases are inefficient and inaccurate in the data query process, especially when faced with complex queries, they consume too much computing resources and are difficult to estimate the amount of target data quickly and accurately.
By extracting keyword features from query statements in a distributed database, keyword density features are generated, and a cardinality estimation model is used to estimate the data cardinality. Combined with historical data distribution, the query process is optimized.
It improves the accuracy and efficiency of data querying, reduces the consumption of computing resources in data tables, and achieves more accurate estimation of the target data quantity.
Smart Images
Figure CN117743381B_ABST
Abstract
Description
Technical Field
[0001] This document relates to the field of distributed technology, and in particular to a data query method and apparatus for a distributed database. Background Technology
[0002] With the continuous development and promotion of the Internet, distributed processing technology has been widely used to meet the rapidly growing demand for data storage and management. In particular, distributed databases have been widely used as an important means of distributed processing technology in the field of data storage. A distributed database refers to a database cluster that uses high-speed computer networks to connect multiple physically dispersed data storage nodes to form a logically unified database cluster. Based on the established distributed database, efficient data storage and access can be performed. Summary of the Invention
[0003] This specification provides one or more embodiments of a distributed database data query method, comprising: obtaining a query statement for data querying in the distributed database; extracting features from query keywords contained in the query statement to obtain query keyword features; determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database, and generating keyword density features based on the keyword data distribution; inputting the query keyword features and the keyword density features into a cardinality estimation model to perform data cardinality estimation, thereby obtaining a cardinality estimate of the target data corresponding to the query keywords.
[0004] This specification provides one or more embodiments of a distributed database data query apparatus, comprising: a query statement acquisition module configured to acquire query statements for data querying in the distributed database; a feature extraction module configured to extract features from query keywords contained in the query statement to obtain query keyword features; a density feature generation module configured to determine the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database, and generate keyword density features based on the keyword data distribution; and a data cardinality estimation module configured to input the query keyword features and the keyword density features into a cardinality estimation model to perform data cardinality estimation, and obtain a cardinality estimate of the target data corresponding to the query keywords.
[0005] This specification provides one or more embodiments of a distributed database data query device, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: acquire a query statement for data querying in the distributed database; extract features from query keywords contained in the query statement to obtain query keyword features; determine the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database, and generate keyword density features based on the keyword data distribution; input the query keyword features and the keyword density features into a cardinality estimation model to perform data cardinality estimation, thereby obtaining a cardinality estimate of the target data corresponding to the query keywords.
[0006] This specification provides one or more embodiments of a storage medium for storing computer-executable instructions, which, when executed by a processor, perform the following process: obtaining a query statement for data querying in a distributed database; extracting features from query keywords contained in the query statement to obtain query keyword features; determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database, and generating keyword density features based on the keyword data distribution; inputting the query keyword features and the keyword density features into a cardinality estimation model to perform data cardinality estimation, and obtaining a cardinality estimate of the target data corresponding to the query keywords. Attached Figure Description
[0007] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0008] Figure 1 A schematic diagram illustrating the implementation environment of a data query method for a distributed database provided in one or more embodiments of this specification;
[0009] Figure 2 A flowchart illustrating a data query method for a distributed database provided in one or more embodiments of this specification;
[0010] Figure 3 A schematic diagram of the architecture of a cardinality estimation model provided for one or more embodiments of this specification;
[0011] Figure 4A flowchart illustrating a data query method for a distributed database applied in a real-world data query scenario, provided for one or more embodiments of this specification;
[0012] Figure 5 A schematic diagram of an embodiment of a data query device for a distributed database provided in one or more embodiments of this specification;
[0013] Figure 6 This is a schematic diagram of the structure of a data query device for a distributed database provided in one or more embodiments of this specification. Detailed Implementation
[0014] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.
[0015] The data query method for a distributed database provided in one or more embodiments of this specification is applicable to the distributed database implementation environment. (Refer to...) Figure 1 The implementation environment includes at least:
[0016] Distributed database 101, query system 102 for querying data in distributed database 101, query system 102 includes: feature extraction module 102-1 for extracting features of query keywords, density feature generation module 102-2 for generating keyword density features, and cardinality estimation model 102-3 for estimating data cardinality.
[0017] The distributed database 101 is deployed on a storage device, and the query system 102 runs on a processing device. The storage device and the processing device can be one or more servers, a server cluster composed of several servers, or a cloud server of a cloud computing platform.
[0018] In this implementation environment, during the data query process of the query system 102, for the query statement of the data query in the distributed database 101, the feature extraction module 102-1 extracts the query keyword features of the query keywords contained in the query statement, and the density feature generation module 102-2 generates the keyword density features corresponding to the query keywords. The feature extraction module 102-1 and the density feature generation module 102-2 transmit the query keyword features and keyword density features to the cardinality estimation model 102-3. The query keyword features and keyword density features are input into the cardinality estimation model 102-3 to perform data cardinality estimation. The cardinality estimate value of the target data corresponding to the query keywords output by the cardinality estimation model 102-3 is used as the query result.
[0019] This specification provides one or more embodiments of a data query method for a distributed database, as follows:
[0020] Reference Figure 2 The data query method for a distributed database provided in this embodiment specifically includes steps S202 to S208.
[0021] Step S202: Obtain the query statement for data querying in the distributed database.
[0022] In the process of data querying in a distributed database, data is retrieved by sending query statements, such as SQL query statements submitted through the distributed database query system. In this embodiment, the data query in the distributed database refers to retrieving the quantity of target data within the distributed database; the quantity of target data to be queried is referred to as the cardinality.
[0023] In practice, during the data query process in the distributed database, provided that the distributed database maintains its own historical query records, the current query statement can be matched and verified. If the matching and verification determines that the data query in the historical query records of the distributed database is relatively close to the current query statement, the query result can be returned directly based on the relatively close data query in the historical query records, thereby improving the response efficiency of data query.
[0024] In one optional implementation of this embodiment, after obtaining the query statement currently used for data querying in the distributed database, the query statement is matched and verified in the following manner:
[0025] The query statement is matched and validated based on the historical query record table of the distributed database.
[0026] If the matching verification result contains the target historical query record, then the target historical query record will be output as the query result of the query statement;
[0027] If the matching verification result is empty, then the following step S204 is executed to extract features from the query keywords contained in the query statement to obtain query keyword features.
[0028] The historical query record table records the query inputs and outputs of the distributed database in the past. Each historical query input and output constitutes a historical query record in the historical query record table.
[0029] In the specific execution process, when matching and verifying the query keywords contained in the current query statement with the historical query record table of the distributed database, the matching and verification can be performed from the data range and time dimension of the query keywords. If there are historical query records in the historical query record table that match the current query statement in terms of data range and time dimension, it indicates that the data query of the matching historical query record is relatively close to the data query of the current query statement.
[0030] In one optional implementation of this embodiment, the query keywords are matched and validated based on the historical query record table of the distributed database, including:
[0031] Verify whether there are any historical query records in the historical query record table that have an overlap with the data range contained in the query keywords that meet the overlap condition;
[0032] If it exists, check whether the query interval corresponding to the query time of the historical query record is less than a preset duration threshold. If it is less, use the historical query record as the target historical query record. If it is greater than or equal to, determine that the matching verification result is empty.
[0033] If it does not exist, the matching verification result is empty.
[0034] In this embodiment, the data range contained in the query keyword refers to the numerical range of the corresponding data to be queried contained in the query keyword. For example, in a distributed database storing resident information of a certain region, when querying the number of residents aged 18 to 45, the data range contained in the query keyword of the SQL query statement is [18, 45]. This data range is also called a continuous data range.
[0035] In addition, data range can also refer to the range of data sets containing the corresponding data to be queried that are included in the query keywords. For example, in a distributed database that stores resident information of a certain region, when querying the number of residents whose residence is in region A, region B, and region C, the data range contained in the query keywords of the SQL query statement is [region A, region B, region C]. This kind of data range is also called discrete data range.
[0036] In the process of verifying whether there are historical query records in the historical query record table that meet the overlap condition with the data range contained in the query keyword, when the data range contained in the query keyword is a numerical range, the numerical overlap between the numerical range contained in the query keyword and the numerical range of the corresponding field record in the historical query record table can be calculated. If the calculated numerical overlap is greater than the preset overlap threshold, it is determined that the overlap condition is met; otherwise, if the calculated numerical overlap is less than or equal to the preset overlap threshold, it is determined that the overlap condition is not met.
[0037] Similarly, when the data range contained in the query keywords is a data set range, the data set range contained in the query keywords can be compared with the data set range of the corresponding field records in the historical query record table to calculate the data overlap. If the calculated data overlap is greater than the preset overlap threshold, the overlap condition is determined to be met; otherwise, if the calculated data overlap is less than or equal to the preset overlap threshold, the overlap condition is determined not to be met.
[0038] Step S204: Extract features from the query keywords contained in the query statement to obtain query keyword features.
[0039] In this embodiment, during the feature extraction process of query keywords contained in the query statement, feature extraction can be performed from at least one keyword dimension to obtain query keyword features. Specifically, feature extraction can be performed on the query keywords in the data range dimension to determine the feature expression of the numerical range of the corresponding data to be queried contained in the query keywords; alternatively, feature extraction can be performed on the query keywords in the data table dimension to determine the data table to be queried in the distributed database based on the query keywords; or feature extraction can be performed on the query keywords in the table join relationship dimension to determine the table join relationship between the data tables to be queried in the distributed database. In addition, feature extraction can also be performed on two or three of the three keyword dimensions—data range dimension, data table dimension, and table join relationship dimension—to obtain the corresponding query keyword features.
[0040] In one optional implementation of this embodiment, feature extraction is performed on the query keywords contained in the query statement to obtain query keyword features, including:
[0041] The query data range features are obtained by extracting the query data range from the query keywords.
[0042] Data table features are obtained by extracting keywords from the query keywords.
[0043] The table join characteristics are obtained by extracting table join relationships from the query keywords;
[0044] The query keyword features are obtained by concatenating the data range features, the data table features, and the table join features.
[0045] In practice, extracting table keywords from query keywords involves extracting features from the table keywords of the data tables being queried, which are contained within the query keywords. For example, a distributed database storing resident information for a certain region might contain several tables. If a query needs to be performed on one or more of these tables, and the SQL query keywords contain the table keywords of the tables being queried, then these table keywords are extracted as table features. By extracting these table features, data queries on the distributed database can be constrained to specific tables, thus improving query efficiency.
[0046] Extracting table join relationships from query keywords refers to extracting features from the table join relationships between the tables being queried, contained within the query keywords. For example, in a distributed database storing resident information for a region, there are several tables. If a query needs to be performed on multiple tables that are joined, the SQL query keywords might contain keywords related to the table join relationships. In this case, these join relationship keywords are extracted as table join features. Extracting table join features reduces the computational burden of simultaneously querying multiple tables. Furthermore, it allows for finer-grained differentiation between data queries, thus improving the accuracy of the data query, specifically the accuracy of the cardinality estimation of the target data.
[0047] For example, when performing a data query in a distributed database that stores resident information of a certain region, the query keywords of the SQL query statement include the data range of [18, 45], and the table keywords of the data tables include the table name or table code of the identity information data table and the table name or table code of the residence data table in the distributed database. It also includes the table join relationship between the identity information data table and the residence data table.
[0048] In the process of feature extraction of query keywords in SQL query statements, the data range [18, 45] contained in the query keywords is extracted, and a data range feature vector is constructed based on the extracted data range [18, 45]. The table names or table codes of the identity information data table and the residence data table contained in the query keywords are extracted, and a data table feature vector is constructed based on the extracted table names or table codes. In addition, the table join keywords of the table join relationship between the identity information data table and the residence data table contained in the query keywords are extracted, and a table join feature vector is constructed based on the extracted table join keywords.
[0049] After obtaining the data range feature vector, data table feature vector, and table join feature vector, the data range feature vector, data table feature vector, and table join feature vector are concatenated to obtain the concatenated vector as the keyword feature vector of the overall query keywords in the SQL query statement.
[0050] As described above, the data range contained in the query keywords includes two types: a continuous data range (numerical range) and a discrete data range (data set range). When the data range contained in the query keywords is a discrete data range (data set range), the discrete data range can be converted into a continuous data range. Data range features can then be extracted after the conversion to a continuous data range. Specifically, in an optional implementation of this embodiment, the data range features are obtained by extracting the query data range of the query keywords, including: if the query data range contained in the query keywords is detected to be a discrete data range, the query data range is encoded, and the continuous data range of the query data range is determined as the data range feature based on the encoding result.
[0051] It should be noted that the above three processes—extracting the query data range from the query keywords, extracting data table keywords from the query keywords, and extracting table join relationships from the query keywords—are not limited in execution order. Any one of these processes can be executed first, followed by the other two. Furthermore, to improve processing efficiency and data query response efficiency, a parallel processing approach can be adopted, where the three processes are executed in parallel on different processing threads.
[0052] Step S206: Determine the keyword data distribution corresponding to the query keyword in the historical data distribution of the distributed database, and generate keyword density features based on the keyword data distribution.
[0053] In practice, based on the query statement used to query data in the distributed database, starting from the query keywords contained in the query statement, the keyword data distribution corresponding to the query keywords is determined in the historical data distribution of the distributed database. Furthermore, based on the determined keyword data distribution, keyword density features are generated. The keyword density features reflect the density distribution of query keywords in the distributed database. Combined with the density distribution of query keywords, the distribution of the target data to be queried in the distributed database is reflected.
[0054] In one optional implementation of this embodiment, determining the keyword data distribution corresponding to the query keyword in the historical data distribution of the distributed database includes:
[0055] Determine the keyword type corresponding to the query keywords;
[0056] The keyword density histogram corresponding to the keyword type is queried from the historical data distribution to serve as the keyword data distribution.
[0057] Specifically, in determining the keyword data distribution corresponding to the query keywords contained in the query statement, in addition to the above-mentioned implementation method of querying the keyword density histogram corresponding to the keyword type in the historical data distribution of the distributed database, other forms of data distribution data can also be queried in the historical data distribution as keyword data distribution.
[0058] Taking an SQL query statement for querying data in a distributed database storing resident information of a certain region as an example, if the keyword type of the query keyword in the SQL query statement is age keyword type, then the age density histogram representing the age distribution will be queried in the historical density histogram of the distributed database. The horizontal axis of the age density histogram is the value of each age, and the vertical axis is the resident count corresponding to each age value.
[0059] Based on the keyword density histogram corresponding to the keyword type determined in the historical data distribution, the keyword density feature is generated in the process of generating keyword density features based on the determined keyword data distribution through the following optional implementation method: performing feature transformation on the keyword density histogram, and using the transformed density histogram features as the keyword density features.
[0060] It should be noted that the above-described process of extracting features from the query keywords contained in the query statement to obtain query keyword features, and the above-described process of determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database and generating keyword density features, are not limited in execution order. The process of extracting features from the query keywords contained in the query statement to obtain query keyword features can be performed first, followed by the process of determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database and generating keyword density features. Alternatively, the process of determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database and generating keyword density features can be performed first, followed by the process of extracting features from the query keywords contained in the query statement to obtain query keyword features. Furthermore, a parallel processing approach can be adopted, with different processing threads performing both the process of extracting features from the query keywords contained in the query statement to obtain query keyword features and the process of determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database and generating keyword density features in parallel.
[0061] Step S208: Input the query keyword features and the keyword density features into the cardinality estimation model to perform data cardinality estimation and obtain the cardinality estimate of the target data corresponding to the query keyword.
[0062] The above-mentioned approach improves the accuracy and efficiency of data queries by extracting features from the query keywords contained in the query statement. It also improves the accuracy of data queries by determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database and generating keyword density features. Based on the obtained query keyword features and keyword density features, data cardinality estimation can be performed by inputting the query keyword features and keyword density features into the cardinality estimation model. This allows for a more comprehensive and accurate cardinality estimation, thereby obtaining a more accurate cardinality estimate of the target data corresponding to the query keywords.
[0063] In this embodiment, the data query performed in the distributed database refers to querying the quantity of target data in the distributed database. Here, the quantity of target data to be queried is referred to as the cardinality. The cardinality estimate of the target data corresponding to the query keywords output by the cardinality estimation model refers to the estimated quantity of target data output by the cardinality estimation model after estimating the target data to be queried.
[0064] In the specific execution process, when the cardinality estimation model estimates the data cardinality based on the input query keyword features and keyword density features, the query keyword features and keyword density features are different types of features. At the same time, the input query keyword features may be one-dimensional features, while the keyword density features may be two-dimensional features. The query keyword features and keyword density features can be unified through feature standardization, and then the data cardinality is estimated based on the two parts of features obtained by feature standardization.
[0065] Optionally, the cardinality estimation model includes: a first perceptron module, a convolutional neural network, and a second perceptron module;
[0066] Based on the cardinality estimation model, which includes a first perceptron module, a convolutional neural network, and a second perceptron module, the first perceptron module can be used to standardize the query keyword features, and the convolutional neural network can be used to standardize the keyword density features. Then, the two parts of features obtained by feature standardization are input into the second perceptron module for data cardinality estimation.
[0067] Specifically, in one optional implementation method provided in this embodiment, the data cardinality estimation is achieved in the following way:
[0068] The first perceptron module performs feature standardization on the query keyword features to obtain standard keyword features;
[0069] The keyword density features are standardized using the convolutional neural network to obtain standard density features.
[0070] The standard keyword features and the standard density features are input into the second perceptron module to estimate the cardinality of the target data corresponding to the query keyword, and the cardinality estimate is obtained.
[0071] For example, in Figure 3 In the cardinality estimation model shown, the keyword feature vector of the entire query keywords in the SQL query statement is input into a multilayer perceptron (MLP) for feature standardization to obtain a standardized keyword feature vector. The keyword density feature vector obtained by feature transformation of the keyword density histogram is input into a convolutional neural network (CNN) for feature standardization to obtain a standardized keyword density feature vector. The standardized keyword feature vector and the standardized keyword density feature vector are then input into an enhanced multilayer perceptron to estimate the number of residents to be queried and output the estimated number of residents.
[0072] In this embodiment, if the distributed database maintains a historical query record table that records its own historical data queries, the cardinality estimation model can also be trained based on the historical query records in the historical query record table that record historical data queries performed in the distributed database. Specifically, in one optional implementation of this embodiment, the cardinality estimation model is trained in the following way: training samples and corresponding sample labels are constructed based on the historical query records in the historical query record table of the distributed database; the model to be trained is trained based on the training samples and the sample labels, and the cardinality estimation model is obtained after training is completed.
[0073] In summary, the data query method for one or more distributed databases provided in this embodiment improves the efficiency of data querying in the distributed database by starting with the query keywords contained in the query statement and estimating the cardinality of the target data through a cardinality estimation model, rather than consuming a large amount of computing resources to query the data tables in the distributed database. Specifically, during the query process, feature extraction is performed on the query keywords contained in the query statement to obtain query keyword features, thereby improving the accuracy and efficiency of data query. Furthermore, the accuracy of data query is improved by determining the keyword data distribution corresponding to the query keywords in the historical data distribution of the distributed database and generating keyword density features. Based on the obtained query keyword features and keyword density features, the cardinality is estimated more comprehensively and accurately by inputting the query keyword features and keyword density features into the cardinality estimation model, thereby obtaining a more accurate cardinality estimate of the target data corresponding to the query keywords.
[0074] The following example uses the application of a distributed database data query method provided in this embodiment in a real-world data query scenario, combined with... Figure 4 The data query method for the distributed database provided in this embodiment will be further explained below. Figure 4 The data query method for distributed databases applied to real-world data query scenarios includes the following steps.
[0075] Step S402: Obtain the SQL query statement for data querying in the distributed database.
[0076] Step S404: Extract the data range feature vector by analyzing the query keywords contained in the SQL query statement.
[0077] Step S406: Extract data table feature vectors from the query keywords contained in the SQL query statement.
[0078] Step S408: Extract table join relationships from the query keywords contained in the SQL query statement to obtain table join feature vectors.
[0079] Step S410: Perform feature concatenation on the data range feature vector, data table feature vector, and table join feature vector to obtain the query keyword feature vector.
[0080] Step S412: Query the keyword density histogram corresponding to the keyword type of the query keyword in the historical data distribution, and perform feature transformation on the keyword density histogram to obtain the keyword density feature vector.
[0081] Step S414: The first perceptron module of the cardinality estimation model is used to standardize the feature vector of the query keywords to obtain the standard keyword feature vector.
[0082] Step S416: The keyword density feature vector is standardized by using a convolutional neural network of a cardinality estimation model to obtain a standard density feature vector.
[0083] Step S418: Input the standard keyword feature vector and the standard density feature vector into the second perceptron module of the cardinality estimation model to perform cardinality estimation of the target data corresponding to the query keywords, and output the cardinality estimate value.
[0084] This specification provides an embodiment of a data query device for a distributed database as follows:
[0085] In the above embodiments, a data query method for a distributed database is provided. Correspondingly, a data query device for a distributed database running on a server is also provided, which will be described below with reference to the accompanying drawings.
[0086] Reference Figure 5 This illustration shows a schematic diagram of an embodiment of a distributed database data query device provided in this embodiment.
[0087] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.
[0088] This embodiment provides a data query device for a distributed database, which runs on a server. The device includes:
[0089] The query statement retrieval module 502 is configured to retrieve the query statement used to query data in a distributed database.
[0090] The feature extraction module 504 is configured to extract features from the query keywords contained in the query statement to obtain query keyword features;
[0091] The density feature generation module 506 is configured to determine the keyword data distribution corresponding to the query keyword in the historical data distribution of the distributed database, and generate keyword density features based on the keyword data distribution;
[0092] The data cardinality estimation module 508 is configured to input the query keyword features and the keyword density features into the cardinality estimation model to perform data cardinality estimation and obtain the cardinality estimate of the target data corresponding to the query keyword.
[0093] This specification provides an example of a data query device for a distributed database as follows:
[0094] Corresponding to the data query method for a distributed database described above, based on the same technical concept, one or more embodiments of this specification also provide a data query device for a distributed database, which is used to execute the data query method for a distributed database provided above. Figure 6 This is a schematic diagram of the structure of a data query device for a distributed database provided in one or more embodiments of this specification.
[0095] This embodiment provides a data query device for a distributed database, comprising:
[0096] like Figure 6 As shown, the data query device for a distributed database can vary significantly due to differences in configuration or performance. It may include one or more processors 601 and a memory 602, where one or more application programs or data can be stored. The memory 602 can be temporary or persistent storage. The application programs stored in the memory 602 may include one or more modules (not shown), each module including a series of computer-executable instructions from the distributed database's data query device. Furthermore, the processor 601 may be configured to communicate with the memory 602, executing the series of computer-executable instructions in the memory 602 on the distributed database's data query device. The distributed database's data query device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, one or more keyboards 606, etc.
[0097] In one specific embodiment, the data query device for a distributed database includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the data query device for the distributed database, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:
[0098] Retrieve the query statement for data retrieval in a distributed database;
[0099] Feature extraction is performed on the query keywords contained in the query statement to obtain query keyword features;
[0100] The keyword data distribution corresponding to the query keyword is determined from the historical data distribution of the distributed database, and a keyword density feature is generated based on the keyword data distribution;
[0101] The query keyword features and the keyword density features are input into the cardinality estimation model to perform data cardinality estimation, thereby obtaining the cardinality estimate of the target data corresponding to the query keyword.
[0102] This specification provides an example of a storage medium as follows:
[0103] Corresponding to the data query method for a distributed database described above, and based on the same technical concept, one or more embodiments of this specification also provide a storage medium.
[0104] The storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process:
[0105] Retrieve the query statement for data retrieval in a distributed database;
[0106] Feature extraction is performed on the query keywords contained in the query statement to obtain query keyword features;
[0107] The keyword data distribution corresponding to the query keyword is determined from the historical data distribution of the distributed database, and a keyword density feature is generated based on the keyword data distribution;
[0108] The query keyword features and the keyword density features are input into the cardinality estimation model to perform data cardinality estimation, thereby obtaining the cardinality estimate of the target data corresponding to the query keyword.
[0109] It should be noted that the embodiment of a storage medium in this specification and the embodiment of a data query method for a distributed database in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.
[0110] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, please refer to each other. Each embodiment focuses on describing the differences from other embodiments. For example, the device embodiment, equipment embodiment, and storage medium embodiment are all similar to the method embodiment, so the description is relatively simple. For reading the relevant content of the device embodiment, equipment embodiment, and storage medium embodiment, please refer to the description of the method embodiment.
[0111] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0112] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.
[0113] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0114] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.
[0115] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.
[0116] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0117] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0120] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0121] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0122] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0123] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising at least one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0124] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0125] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.
Claims
1. A data query method for a distributed database, comprising: Retrieve the query statement for data retrieval in a distributed database; Feature extraction is performed on the query keywords contained in the query statement to obtain query keyword features; Determine the keyword type corresponding to the query keyword, query the keyword density histogram corresponding to the keyword type in the historical data distribution of the distributed database as the keyword data distribution, and generate keyword density features based on the keyword data distribution; The query keyword features and the keyword density features are input into the cardinality estimation model to perform data cardinality estimation, thereby obtaining the cardinality estimate of the target data corresponding to the query keyword.
2. The data query method for a distributed database according to claim 1, wherein the step of extracting features from the query keywords contained in the query statement to obtain query keyword features includes: The query data range features are obtained by extracting the query data range from the query keywords. Data table features are obtained by extracting keywords from the query keywords. The table join characteristics are obtained by extracting table join relationships from the query keywords; The query keyword features are obtained by concatenating the data range features, the data table features, and the table join features.
3. The data query method for a distributed database according to claim 2, wherein the step of extracting the data range features from the query keywords includes: If the query data range contained in the query keyword is detected to be a discrete data range, the query data range is encoded, and the continuous data range of the query data range is determined as the data range feature based on the encoding result.
4. The data query method for a distributed database according to claim 1, wherein generating keyword density features based on the keyword data distribution includes: The keyword density histogram is subjected to feature transformation, and the transformed density histogram features are used as the keyword density features.
5. The data query method for a distributed database according to claim 1, wherein the cardinality estimation model comprises: First perceptron module, convolutional neural network, and second perceptron module; The data cardinality estimation is achieved in the following manner: The first perceptron module performs feature standardization on the query keyword features to obtain standard keyword features; The keyword density features are standardized using the convolutional neural network to obtain standard density features. The standard keyword features and the standard density features are input into the second perceptron module to estimate the cardinality of the target data corresponding to the query keyword, and the cardinality estimate is obtained.
6. The data query method for a distributed database according to claim 1, wherein the cardinality estimation model is trained in the following manner: Training samples and corresponding sample labels are constructed based on historical query records in the historical query record table of the distributed database. The model to be trained is trained based on the training samples and the sample labels, and the cardinality estimation model is obtained after training is completed.
7. The data query method for a distributed database according to claim 1, further comprising, after the step of obtaining the query statement for data querying in the distributed database is executed, and before the step of extracting features from the query keywords contained in the query statement to obtain query keyword features, the method further comprises: The query keywords are matched and validated based on the historical query record table of the distributed database. If the matching verification result contains the target historical query record, then the target historical query record will be output as the query result of the query statement; If the matching verification result is empty, then the step of extracting features from the query keywords contained in the query statement to obtain query keyword features is performed.
8. The data query method for a distributed database according to claim 7, wherein the step of matching and validating the query keywords based on the historical query record table of the distributed database includes: Verify whether there are any historical query records in the historical query record table that have an overlap with the data range contained in the query keywords that meet the overlap condition; If it exists, check whether the query interval duration corresponding to the query time of the historical query record is less than a preset duration threshold. If it is less, use the historical query record as the target historical query record. If it does not exist, the matching verification result is empty.
9. A data query device for a distributed database, comprising: The query statement retrieval module is configured to retrieve the query statements used to query data in a distributed database. The feature extraction module is configured to extract features from the query keywords contained in the query statement to obtain query keyword features; The density feature generation module is configured to determine the keyword type corresponding to the query keyword, query the keyword density histogram corresponding to the keyword type in the historical data distribution of the distributed database as the keyword data distribution, and generate keyword density features based on the keyword data distribution. The data cardinality estimation module is configured to input the query keyword features and the keyword density features into the cardinality estimation model to perform data cardinality estimation and obtain the cardinality estimate of the target data corresponding to the query keyword.
10. A data query device for a distributed database, comprising: processor; And, a memory configured to store computer-executable instructions, which, when executed, cause the processor to: Retrieve the query statement for data retrieval in a distributed database; Feature extraction is performed on the query keywords contained in the query statement to obtain query keyword features; Determine the keyword type corresponding to the query keyword, query the keyword density histogram corresponding to the keyword type in the historical data distribution of the distributed database as the keyword data distribution, and generate keyword density features based on the keyword data distribution; The query keyword features and the keyword density features are input into the cardinality estimation model to perform data cardinality estimation, thereby obtaining the cardinality estimate of the target data corresponding to the query keyword.
11. A storage medium for storing computer-executable instructions, which, when executed by a processor, perform the following process: Retrieve the query statement for data retrieval in a distributed database; Feature extraction is performed on the query keywords contained in the query statement to obtain query keyword features; Determine the keyword type corresponding to the query keyword, query the keyword density histogram corresponding to the keyword type in the historical data distribution of the distributed database as the keyword data distribution, and generate keyword density features based on the keyword data distribution; The query keyword features and the keyword density features are input into the cardinality estimation model to perform data cardinality estimation, thereby obtaining the cardinality estimate of the target data corresponding to the query keyword.