Data query method, query engine, device, medium and program product

Abstract

The embodiment of the invention provides a data query method, a query engine, equipment, a medium and a program product, and the method comprises the following steps: after a query request is initiated to a data lake, firstly reading global statistical information of the data lake, then determining a query strategy corresponding to the query request according to the global statistical information, and sending the query strategy to a server; and finally, the query request is responded according to the query strategy. Wherein the global statistical information of the data lake is a statistical result obtained after statistics of all data in the data lake. Correspondingly, the local statistical information of the data lake is statistical information obtained by performing statistics on data contained in the data partitions in the data lake. Therefore, compared with the local statistical information of the data lake, the global statistical information of the data lake contains less data volume, so that the query strategy is determined by directly reading the global statistical information, the data reading amount of the query equipment can be reduced, the determination speed of the query strategy is improved, and the query speed of the data in the data lake is further improved.

Description

Technical Field

[0001] The present invention relates to the field of data lake technology, and in particular to a data query method, query engine, device, medium, and program product. Background Art

[0002] A data lake is an architecture for storing and processing large-scale, diverse data. Similar to a database, a data lake can store data in the form of tables. In practice, query engines can query data on the lake using statistical information. This statistical information can be compiled after the data is stored on the lake.

[0003] Based on the above description, how to improve the query speed of lake data becomes an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data query method, query engine, device, medium, and program product to improve the query speed of lake data.

[0005] In a first aspect, an embodiment of the present invention provides a data query method, comprising:

[0006] Responding to a query request from the data lake, reading global statistical information of the data lake;

[0007] Determining a query strategy corresponding to the query request according to the global statistical information;

[0008] Respond to the query request according to the query strategy.

[0009] In a second aspect, an embodiment of the present invention provides a query engine, comprising: a processing component and a query optimization component;

[0010] The processing component is configured to respond to a query request from the data lake by reading global statistical information of the data lake; and respond to the query request according to the query strategy determined by the query optimization component;

[0011] The query optimization component is configured to determine a query strategy corresponding to the query request according to the global statistical information.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory configured to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the data query method described in the first aspect. The electronic device may also include a communication interface configured to communicate with other devices or communication systems.

[0013] In a fourth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a computing system of an electronic device, the computing system can at least implement the data query method described in the first aspect.

[0014] In a fifth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the processor is enabled to implement the data query method as described in the first aspect.

[0015] The data query method provided by an embodiment of the present invention can first read the global statistical information of the data lake after initiating a query request to the data lake, then determine the query strategy corresponding to the query request based on the global statistical information, and finally respond to the query request according to the query strategy. Among them, the global statistical information is the statistical result obtained after counting all the data in the data lake, which is used to describe the overall characteristics of the data in the data lake. Correspondingly, the local statistical information of the data lake is the statistical information obtained after counting the data contained in the data partitions in the data lake, which is used to describe the local characteristics of the data in the data lake. The statistical information corresponding to different data partitions can be collectively referred to as the local statistical information of the data lake.

[0016] It can be seen that compared with the local statistical information of the data lake, the global statistical information of the data lake contains less data. Therefore, by directly reading the global statistical information to determine the query strategy, the amount of data read can be reduced, thereby increasing the speed of determining the query strategy and the speed of querying data in the data lake. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a data query method provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of a statistical information merging method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a process for writing characteristic values ​​of a data partition provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of a data structure storing characteristic values ​​of a data lake provided by an embodiment of the present invention;

[0022] Figure 5 A structural diagram of a query engine provided by an embodiment of the present invention;

[0023] Figure 6 A schematic structural diagram of a data query device provided by an embodiment of the present invention;

[0024] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0026] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a," "the," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. "A plurality" generally includes at least two, but does not exclude the inclusion of at least one.

[0027] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0028] As used herein, the words “if” and “if” may be interpreted as “at the time of” or “when” or “in response to determining” or “in response to identifying,” depending on the context. Similarly, the phrases “if it is determined” or “if (stated condition or event) is identified” may be interpreted as “when it is determined” or “in response to determining” or “when identifying (stated condition or event)” or “in response to identifying (stated condition or event),” depending on the context.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or system comprising the element.

[0031] Before describing the methods and engines provided by the following embodiments of the present invention, the following concepts may be explained:

[0032] Data lake: An architecture for storing and processing large-scale, diverse data. Within a data lake, data can be stored in tables. Compared to databases, data lakes have lower requirements for data structure.

[0033] Data partition: A table in a data lake can be divided into at least one sub-table, each of which is a data partition. A data partition in a data lake contains at least one field.

[0034] Local statistics of a data lake: Statistics are calculated on a field-by-field basis, taking the data in any data partition of the data lake as a whole. This statistical result is the statistical information for that data partition. A data lake can contain multiple data partitions, and the statistical information for each of these partitions is collectively referred to as the local statistics of the data lake. Local statistics can be used to describe the local characteristics of the data in the data lake. Global statistics of a data lake: Statistics are calculated on a field-by-field basis, taking the data in all data partitions of the data lake as a whole. This statistical result is the global statistics of the data lake. Global statistics can be used to describe the overall characteristics of the data in the data lake.

[0035] Optionally, both local statistical information and global statistical information can include at least one of the following items: the total number of rows, the amount of data, the maximum value in a field, the null rate (Null Rate) of a field, and the number of different values ​​(NDV) in a field, that is, the field cardinality of the field.

[0036] Global statistics can include: the total number of rows, data volume, maximum value, null value ratio, field cardinality, etc. for each field in all data partitions within the data lake. Local statistics can also include: the total number of rows, data volume, maximum value, null value ratio, field cardinality, etc. for each field in each data partition within the data lake.

[0037] As can be seen, global statistical information can be a set of statistical data obtained by analyzing all data in the data lake. This set of statistical data can include at least one of the items mentioned above. Local statistical information can be a set of statistical data for any data partition in the data lake, obtained by analyzing the data contained in that partition. This set of statistical data can also include at least one of the items mentioned above.

[0038] In practice, since a data lake contains multiple data partitions, the local statistical information of the data lake can include a set of statistical information for each of the multiple data partitions, that is, it includes multiple sets of statistical data. Therefore, the amount of data in the global statistical information is much smaller than that in the local statistical information.

[0039] Query engine: A component used to query data in the data lake. The query engine can also perform statistics on the data in the data lake to obtain local and global statistical information for the data lake.

[0040] Based on the above introduction, some embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the absence of conflicts between the embodiments, the following embodiments and the features in the embodiments may be combined with each other. In addition, the step timings in the following method embodiments are only examples and are not strictly limiting.

[0041] Figure 1 This is a flow chart of a data query method provided by an embodiment of the present invention. The method provided by an embodiment of the present invention can be executed by a query device. More precisely, the method can be executed by a query engine deployed in the query device. Figure 1 As shown, the method may include the following steps:

[0042] S101, in response to a query request of the data lake, read the global statistical information of the data lake.

[0043] S102: Determine a query strategy corresponding to the query request according to global statistical information.

[0044] S103: respond to the query request according to the query strategy.

[0045] A user can generate a query request for the data lake. In response to the request, the query device can first read the global statistical information of the data lake, then determine the query strategy corresponding to the query request based on the global statistical information. Finally, the query device can execute this query strategy to respond to the query request.

[0046] As described above, global statistics can be the result of analyzing all data in a data lake. Correspondingly, local statistics for a data lake can be the result of analyzing the data contained in a data partition within the data lake. The statistics corresponding to different data partitions within a data lake are collectively referred to as the local statistics of the data lake. In practice, due to the large number of data partitions in a data lake, the data volume of local statistics is much larger than that of global statistics.

[0047] As described above, both local statistical information and global statistical information can include at least one of the following items: the total number of rows in a column (i.e., a field), the amount of data, the maximum value, the null rate (Null Rate), and the number of different values ​​in a column (Number of Different Values, abbreviated as NDV), which is also the field cardinality.

[0048] Optionally, the query device can merge the statistical information of different data partitions in the data lake and determine the merged result as the global statistical information of the data lake. Optionally, for any data partition in the data lake, the statistical information of the data partition, that is, the local statistical information of the data lake, can be generated by the query device.

[0049] Optionally, the query device can use different operators to determine a query strategy that includes a table join order and data propagation method. The join order can be determined by the join operator, while the data propagation method can be determined by the exchange operator. The execution costs of the operators can then be estimated based on global statistical information from the data lake, ultimately determining the table join order and data propagation method that minimizes the execution cost, thereby determining the query strategy.

[0050] The query device then joins the multiple tables involved in the query request according to the table join order specified in the query strategy to obtain candidate data. It then uses the query conditions specified in the query request to retrieve the target data that meets the query conditions from the candidate data. The target data is then fed back to the user as a response to the query request using the data propagation method specified in the query strategy, effectively completing the query request response.

[0051] The generation and merging methods of local statistical information may refer to the description in the following related embodiments. Optionally, the merging of statistical information may be performed periodically.

[0052] The following example illustrates the relationship between global and local statistics:

[0053] Assume that the data lake contains three data partitions, namely data partition 1 to data partition 3, and the data lake can store data with the two fields "user name" and "height": [User 1, height 150cm], [User 2, height 170cm], [User 3, height 160cm], [User 4, height 150cm], [User 5, height 180cm], [User 6, height 175cm], [User 7, height 180cm], [User 8, height 185cm], [User 9, height 160cm].

[0054] The above data can be stored in three data partitions respectively. For example, data partition 1 may contain the following data: [User 1, height 150 cm], [User 2, height 170 cm], [User 3, height 160 cm]. Data partition 2 may contain the following data: [User 4, height 150 cm], [User 5, height 180 cm], [User 6, height 150 cm]. Data partition 3 may contain the following data: [User 7, height 180 cm], [User 8, height 185 cm], [User 9, height 160 cm].

[0055] Taking the "Height" field as an example, the statistical information for data partition 1 might include: maximum value 170cm, minimum value 150cm, total number of rows 3, and field cardinality 3. The statistical information for data partition 2 might include: maximum value 180cm, minimum value 150cm, total number of rows 3, and field cardinality 2. The statistical information for data partition 3 might include: maximum value 185cm, minimum value 160cm, total number of rows 3, and field cardinality 3.

[0056] The query device can merge the statistical information of the "height" field above. The final global statistical information of the data lake is: the maximum value is 185cm, the minimum value is 150cm, the total number of rows is 9, and the field cardinality is 5.

[0057] In this embodiment, after initiating a query request to the data lake, the global statistical information of the data lake can be read first. Then, based on the global statistical information, a query strategy corresponding to the query request is determined, and finally, the query request is responded to according to this query strategy. Because the global statistical information of the data lake contains less data than the local statistical information of the data lake, determining the query strategy by directly reading the global statistical information can reduce the amount of data read by the query device, thereby speeding up the query strategy determination and further improving the efficiency of data queries in the data lake.

[0058] In addition, the technical effects that can be achieved by the data query methods provided by various embodiments of the present invention can also be understood in conjunction with the following:

[0059] In practice, local statistical information of the data lake can also be used to respond to query requests. The specific process can be:

[0060] In response to a query request, the query device may first obtain target statistical information related to the query request from the local statistical information of the data lake, and then merge this target statistical information. The query device may then determine a query strategy based on the merged results and respond to the query request according to this query strategy. The target statistical information may be statistical information for the target data partition related to the query request.

[0061] When using this approach, after a query request is generated, the query device needs to determine and merge the target statistical information in real time. This real-time statistical process obviously slows down the query strategy determination process, which in turn reduces the query response speed. Furthermore, the more target data partitions associated with a query request, the longer it takes for the query device to merge the statistical information, which also affects the query response speed.

[0062] Compared to the above approach, the embodiments of the present invention propose global statistical information that can be used in data lake scenarios. Furthermore, with the help of this global statistical information, the query device can omit the process of merging local statistical information during the query request response process, thereby increasing the speed of determining the query strategy and further increasing the response speed of the query request. Furthermore, because the query device directly determines the query strategy based on the global statistical information, the response speed of the query request is not affected regardless of the number of target data partitions involved in the query request. In other words, the use of global statistical information ensures that the number of target data partitions does not affect the response speed of the query request.

[0063] Optionally, the statistical information of the data lake can be stored as metadata of the data in the data lake in a database (Metastore). Optionally, the database can be independent of the data lake.

[0064] Optionally, when the database stores global statistics for the data lake, the query device can leverage its own caching mechanism to improve query response speed. Specifically, upon receiving a query request, the query device can prioritize reading the global statistics for the data lake from its local cache. If the query device's local cache lacks global statistics, it reads them from the database and stores them in its local cache.

[0065] Based on the caching mechanism, the technical effects of using global statistical information for data query can also be understood in conjunction with the following:

[0066] Compared with local statistical information, global statistical information has a smaller data volume and occupies a smaller storage space. Therefore, compared with local statistical information, global statistical information can be more completely stored in the cache of the query device. This can improve the hit rate of information in the cache during the query request response process. That is, when responding to a query request, the query device is likely to read the global statistical information from the cache, thereby improving the response speed of the query request.

[0067] Optionally, the database can be shared by multiple data lakes. In this case, the technical effects of using global statistical information for data query can also be understood in conjunction with the following:

[0068] When a database is shared by multiple data lakes and stores local statistical features of each data lake, and a query request is sent to a target data lake among the multiple data lakes, and the number of target data partitions associated with the query request is large, the query device corresponding to the target data lake needs to read a large amount of local statistical information from the database to respond to the query request. Reading large amounts of local statistical information consumes a large amount of resources, which may prevent the query devices corresponding to other data lakes from reading the local statistical information of other data lakes from the database, causing the other data lakes to be unusable.

[0069] In the embodiments provided by the present invention, even if there are a large number of target data partitions related to the query request, the query device can directly use the global statistical information of the target data lake with a small amount of data read from the database to respond to the query request. Therefore, the above-mentioned situation of affecting the normal use of other data lakes will not occur.

[0070] Figure 1 In the embodiment shown, it has been mentioned that the query device can obtain the global statistical information of the data lake by merging local statistical information. Figure 1As can be seen from the examples in the illustrated embodiments, the total number of rows and the maximum value in the statistical information can be directly merged, that is, the total number of rows in the local statistical features can be directly added to obtain the total number of rows in the global statistical information, and the maximum value in the local statistical information can be compared to obtain the maximum value in the global statistical information.

[0071] However, the field cardinality in the statistical information cannot be directly obtained by simple calculation. In this case, the following embodiment can be used to realize the merging of the field cardinality in the local statistical features. Figure 2 This is a flow chart of a statistical information merging method provided by an embodiment of the present invention. The method provided by an embodiment of the present invention can also be executed by a query device. Figure 2 As shown, the method may include the following steps:

[0072] S201 , performing statistics on characteristic values ​​of different data partitions stored in the first data structure to obtain characteristic values ​​of a data lake stored in the form of a second data structure.

[0073] S202: Convert the storage format of the characteristic values ​​of the data lake from the second data structure to the first data structure.

[0074] S203: Determine the field cardinality of the data lake according to the characteristic values ​​of the data lake stored in the first data structure.

[0075] Because data in a data lake is stored in partitions, data on the lake is written to a specific data partition within the data lake. After data is written to a data partition, the query device can first obtain the characteristic values ​​of each data partition. Furthermore, by statistically analyzing these characteristic values, the characteristic values ​​of the data lake can be obtained. Finally, the field cardinality of the data lake is calculated based on the characteristic values ​​of the data lake.

[0076] The characteristic values ​​of data partitions can be generated by the query device when data partitions are added or deleted. The characteristic values ​​of data partitions and the data lake can be stored in corresponding data structures, such as tables, queues, arrays, etc. The characteristic values ​​can be considered the basic data used to calculate field cardinality.

[0077] Optionally, the characteristic values ​​of different data partitions in the data lake can be stored in the first data structure, and the characteristic values ​​of the data lake can be stored in the second data structure. In order to perform calculations on the field data of the data lake, the characteristic values ​​of the data lake can also be converted from the second data structure and stored in the first data structure.

[0078] Optionally, the first data structure may be a one-dimensional array, the second data structure may be a two-dimensional array, and the length of the one-dimensional array is the same as the number of rows of the two-dimensional array. Optionally, the one-dimensional array may be a one-dimensional array of type int, and the two-dimensional array may be a two-dimensional array of type int.

[0079] The following describes how the query device generates a one-dimensional array storing the characteristic values ​​of any data partition:

[0080] For any data partition, the query device can first perform a hash calculation on the data in the partition to obtain a hash calculation result. After the hash calculation, the data in the data partition is converted into binary data. Then, the hash calculation result represented as binary data can be truncated into a first part and a second part as an index. The maximum number of leading zeros in the first part of the hash calculation result can be written to the position pointed to by the second part of the hash calculation result in the one-dimensional array to obtain the one-dimensional array corresponding to the any data partition. Wherein, the hash calculation result can be P-bit binary data, then the first part can be L bits, and the second part can be T bits, where P=L+T. A common setting method is that the binary data is 64 bits, L=54, and T=10.

[0081] The above one-dimensional array generation process can also be combined with Figure 3 Understand. Figure 3 As shown, assuming that the length of the one-dimensional array corresponding to the data partition is M, the elements in the one-dimensional array can be respectively called delta[0] to delta[M-1]. And as Figure 3 The one-dimensional data shown is actually a HyperLogLog data structure.

[0082] The query device can first perform a hash calculation on a piece of data in the data partition. Assume that the hash calculation result of the data is 0010100101101. The hash calculation result can be truncated into a first part "00101101" and a second part "00101" as an index. The number of leading zeros in the first part is 2, and the second part is converted to a decimal value of 5. Then "5" can be used as an index to point to delta[5] in the one-dimensional array, and the number of leading zeros 2 can be written to delta[5] in the one-dimensional array.

[0083] The hash calculation result of another data in any data partition can also be processed in the above manner: if the index value of the other data is also 5, and the number of leading zeros is 4, which is greater than 2, then the value on delta[5] is updated to 4. After performing the above processing on each data in any data partition, the one-dimensional array corresponding to the data partition can be obtained.

[0084] Optionally, before truncating the hash calculation result, the query device may also bucket the hash calculation result, that is, bucket the data in the data partition. Optionally, the bucketing process may be: if the hash calculation result is within the hash value range corresponding to the bucket, then the hash calculation result may be divided into the bucket. The number of buckets in any data partition is the same as the length of the one-dimensional array corresponding to the any data partition. Optionally, the number of buckets may be pre-set, and L may be calculated based on the functional relationship between the preset number of buckets and the above-mentioned L.

[0085] In practice, data belonging to the same bucket often have the same index. Therefore, the one-dimensional array corresponding to any data partition actually records the maximum number of leading zeros corresponding to the hash calculation result of the data in each bucket. This maximum number is also the characteristic value of the data partition.

[0086] In this embodiment, the characteristic value of any data partition in the data lake can be obtained by bucketing, hashing, leading zero calculation, etc. The characteristic value can be stored in an HLL data structure, that is, a one-dimensional array.

[0087] After obtaining a one-dimensional array storing characteristic values ​​of data partitions based on the above method, optionally, this one-dimensional array can be used to generate a two-dimensional array recording characteristic values ​​of the data lake. Figure 4 As shown, assuming that the characteristic values ​​of the data lake can be stored in an M*N two-dimensional array, the meaning of the value stored at any position [i, j] in the two-dimensional array can be defined as: in the one-dimensional array corresponding to each of the different data partitions contained in the data lake, the value stored at delta[i] is the number of j.

[0088] Based on the above definition, the query device can generate a two-dimensional array corresponding to the data lake based on the one-dimensional arrays corresponding to the different data partitions in the data lake. The target elements in the second array are the number of target values ​​in the one-dimensional array. The target elements correspond to the target number of rows and columns in the second array. The position of the target value in the one-dimensional array is the same as the target number of rows, and the target value is the same as the target number of columns.

[0089] In this embodiment, the query device can provide a new data structure for storing characteristic values ​​of the data lake. Based on this data structure, the field cardinality of the data lake can be calculated more accurately.

[0090] Furthermore, when data partitions are added or deleted in the data lake, the field cardinality of the data lake also needs to be updated. One optional update method is to update the one-dimensional array containing the characteristic values ​​corresponding to each data partition, then merge the updated one-dimensional arrays. The merged result can be used to determine the field cardinality of the data lake. Assuming the data lake contains 1 million data partitions and the size of a one-dimensional array is 1KB, the query device needs to read 1 million * 1KB = 1GB of data when updating using this method.

[0091] With the help of Figure 4 Another optional update method for the provided two-dimensional array is that the query device can directly merge the one-dimensional array corresponding to the added or deleted data partition into the two-dimensional array corresponding to the data lake in response to the addition or deletion of the data partition, that is, update the two-dimensional array corresponding to the data lake with the one-dimensional array corresponding to the added or deleted data partition. The query device can ultimately determine the field cardinality of the data lake based on the updated two-dimensional array. More specifically, the query device can, based on preset rules, add the feature values ​​in the one-dimensional array corresponding to the added data partition to the two-dimensional array corresponding to the data lake; or delete the feature values ​​in the one-dimensional array corresponding to the deleted data partition from the two-dimensional array corresponding to the data lake.

[0092] Assume the data lake contains 1 million data partitions, each with 2048 buckets, the first part of the truncated binary data is 54 bits, and int data occupies 4 bytes. Using this update method, the amount of data read by the query device during the merge process is 54 * 2048 * 4 (the size of the two-dimensional array of the data lake) + 1K (the size of the one-dimensional array of the newly added or deleted data partition) = 433K.

[0093] This embodiment provides an incremental method for merging feature values. Furthermore, by updating feature values ​​using a two-dimensional array containing the data lake's feature values, this embodiment significantly reduces the amount of data read by query devices. When data partitions in the data lake are added or deleted, the updated two-dimensional array corresponding to the data lake can be retrieved more quickly.

[0094] The following also details the update process of the two-dimensional array when data partitions are added or deleted.

[0095] When a new data partition is added to the data lake, the query device can obtain the one-dimensional array corresponding to the new data partition and the two-dimensional array corresponding to the data lake in response to the addition of the data partition. The query device can then sequentially traverse the one-dimensional array corresponding to the new data partition and increment the elements corresponding to the first reference row number and the first reference column number in the two-dimensional array corresponding to the data lake by one, where the first reference column number is the same as the first element in the one-dimensional array corresponding to the newly traversed new data partition, and the first reference row number and the first element have the same position in the one-dimensional array corresponding to the newly added data partition.

[0096] The above process can also be understood in conjunction with the following pseudo code:

[0097] for(K->0:M-1){

[0098] gsm[K][A1.get(K)]++;

[0099] }

[0100] Where gsm is the name of the two-dimensional data, A1 is the one-dimensional array corresponding to the newly added data partition, and A1.get(K) is the data stored at the lta[K] position in the one-dimensional array.

[0101] For example, for a one-dimensional array of length M corresponding to a newly added data partition, if K = 1, that is, the feature value at the de lta[1] position in the one-dimensional array currently traversed is W, the query device can add 1 to the value at the [1, W] position in the two-dimensional array. Then, the query device continues to traverse the feature value at the de lta[2] position in the one-dimensional array until it reaches the feature value at the de lta[M-1] position. At this point, the feature value of the newly added data partition is updated to the feature value of the data lake.

[0102] When a data partition is deleted from the data lake, in response to the deletion of the data partition, a one-dimensional array corresponding to the deleted data partition and a two-dimensional array corresponding to the data lake are obtained. The query device can then sequentially traverse the one-dimensional array corresponding to the deleted data partition and decrement the elements corresponding to the second reference row number and the second reference column number in the two-dimensional array corresponding to the data lake by one, where the second reference column number is the same as the second element in the one-dimensional array corresponding to the currently traversed deleted data partition, and the second reference row number and the second element have the same position in the one-dimensional array corresponding to the deleted data partition.

[0103] The above process can also be understood in conjunction with the following pseudo code:

[0104] for(K->0:M-1){

[0105] gsm[K][A2.get(K)]--;

[0106] }

[0107] Where gsm is the name of the two-dimensional data, A2 is the one-dimensional array corresponding to the deleted data partition, and A2.get(K) is the data stored at the de lta[K] position in the one-dimensional array.

[0108] Continuing with the example, for a one-dimensional array of length M corresponding to the deleted data partition, if K=1, that is, the feature value at the de lta[1] position currently traversed in the one-dimensional array is W, the query device can add 1 to the value at the [1, W] position in the two-dimensional array. Then, continue to traverse the feature value at the de lta[2] position in the one-dimensional array until the feature value at the de lta[M-1] position is traversed. At this point, the feature value of the deleted data partition is deleted from the feature value of the data lake.

[0109] In addition, according to Figure 2 As described in the illustrated embodiment, in order to implement the calculation of the field data of the data lake, the characteristic values ​​of the data lake also need to be converted and stored from the second data structure to the first data structure. More specifically, the storage format of the characteristic values ​​of the data lake is converted from a two-dimensional array to a one-dimensional array.

[0110] In an optional conversion method, the query device can sequentially traverse different rows in the two-dimensional array corresponding to the data lake in a preset direction to obtain the first non-zero data in each row. Based on the order in which the non-zero data corresponding to different rows in the two-dimensional array are obtained, the non-zero data corresponding to different rows in the two-dimensional array are written sequentially by the number of columns in the two-dimensional array into the one-dimensional array corresponding to the data lake.

[0111] Optionally, the preset direction can be to traverse forward from the last column of the two-dimensional array. Continuing with the above example, for a one-dimensional array of length M and a two-dimensional array of length M*N, and assuming that it is possible to traverse row by row starting from row 0 of the two-dimensional array, it can be determined through traversal that the first non-zero data in row 0 is in column 5, then the column number "5" of the non-zero data can be written to delta[0] of the one-dimensional array.

[0112] Similarly, by traversing, we can determine that the first non-zero data in the first row of the two-dimensional array is in the eighth column. In this case, we can write the column number "8" of the non-zero data into delta[1] of the one-dimensional array. Similarly, when the first non-zero data in the M-1th row of the two-dimensional array is in the sixth column, we can write the column number "6" into delta[M-1] of the one-dimensional array. The above conversion process can also be understood in conjunction with the following pseudocode:

[0113]

[0114]

[0115] Where gsm is the name of the two-dimensional data, M-1 and N-1 are the maximum number of rows and columns of the two-dimensional array, respectively.

[0116] In this embodiment, the second array storing the characteristic values ​​of the data lake can be converted into the first array, which can make the field cardinality subsequently calculated based on the characteristic values ​​more accurate.

[0117] The above has described the working process of the query device from the perspective of the method. The above process can be specifically executed by the query engine in the query device. The following can also describe the table file merging process from the perspective of the query engine. Figure 5 A structural diagram of a query engine provided by an embodiment of the present invention. Figure 5 As shown, the query engine may include: a processing component and a query optimization component.

[0118] In response to a query request from the data lake, the processing component can read the data lake's global statistical information. The query optimization component can determine a query strategy corresponding to the query request based on the global statistical information, and this query strategy can be sent to the processing component. Ultimately, the processing component can respond to the query request according to the query strategy determined by the query optimization component. Optionally, the processing component can combine the statistical information of different data partitions in the data lake, i.e., the local statistical information of the data lake, to obtain the global statistical information of the data lake.

[0119] In this embodiment, after initiating a query request to the data lake, the processing component can first read the data lake's global statistical information. The query optimization component can then determine the query strategy corresponding to the query request based on the global statistical information. Ultimately, the processing component can respond to the query request according to this query strategy. Because the data lake's global statistical information contains less data than the data lake's local statistical information, the query optimization component uses the global statistical information read by the processing component to determine the query strategy. This can reduce the amount of data the processing component needs to read, thereby speeding up the query strategy determination process and further improving the efficiency of data queries in the data lake.

[0120] In addition, compared to using the global statistics of the data lake to respond to query requests, Figure 5In the illustrated embodiment, when the query engine uses the data lake's global statistical information to respond to query requests, it can omit the process of merging the target statistical information of the target data partitions corresponding to the query request, thereby speeding up the determination of query strategies and further improving the response speed of query requests. Furthermore, because the query engine directly determines query strategies based on global statistical information, the response speed of the query request is not affected by the number of target data partitions involved. In other words, the use of global statistical information ensures that the number of target data partitions does not affect the response speed of the query request.

[0121] In addition, for the contents not described in detail in this embodiment and the technical effects that can be achieved, please refer to the relevant descriptions in the above embodiments and will not be repeated here.

[0122] Optionally, the statistical information of the data lake can be stored as metadata of the data in the data lake in a database (Metastore). Optionally, the database can be independent of the data lake.

[0123] Optionally, when the database stores global statistical information of the data lake, the query device can also use its own cache mechanism to improve the response speed of the query request. For details, please refer to the relevant description in the above embodiment and will not be repeated here.

[0124] Alternatively, the database can be shared by multiple data lakes. The technical benefits of using global statistical information for data queries can also be understood in conjunction with the following:

[0125] When a database is shared by multiple data lakes and stores local statistical features of the data lakes, and a query request is directed to a target data lake among the multiple data lakes and the number of target data partitions associated with the query request is large, the query device can directly respond to the query request using the global statistical information of the target data lake, which has a smaller amount of data, read from the database. Therefore, the aforementioned situation of affecting the normal use of other data lakes will not occur. For related details, please refer to the description of the above embodiment and will not be repeated here.

[0126] Optionally, Figure 5 In the embodiment shown, it has been mentioned that the processing component in the query engine can obtain the global statistical information of the data lake by merging the local statistical information. Figure 1 As can be seen from the examples in the embodiment shown, the total number of rows and the maximum value in the statistical information can be directly merged. However, the field cardinality in the statistical information cannot be directly merged. The processing component can also use Figures 2 to 4 The method in the illustrated embodiment obtains the field cardinality in the global feature information.

[0127] Among them, the characteristic data of the data partition used in the field cardinality merging process can also be generated by the processing component in the query engine, and the update of the field cardinality can also be performed by the processing component. The specific process can be found in the description in the above-mentioned related embodiments and will not be repeated here.

[0128] In addition, for the contents not described in detail in this embodiment and the technical effects that can be achieved, please refer to the relevant descriptions in the above embodiments and will not be repeated here.

[0129] The data query device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these data query devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0130] Figure 6 A schematic diagram of a data query device according to an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device may include:

[0131] The reading module 11 is configured to read global statistical information of the data lake in response to a query request of the data lake.

[0132] The strategy determination module 12 is configured to determine a query strategy corresponding to the query request according to the global statistical information.

[0133] The response module 13 is configured to respond to the query request according to the query strategy.

[0134] The method is applied to a query engine, the global statistical information of the data lake is stored in a database shared by multiple data lakes, or in a cache of a query device, and the query engine is deployed in the query device.

[0135] Optionally, the apparatus further includes: a global statistical information determination module 14, configured to determine global statistical information of the data lake based on respective statistical information of different data partitions in the data lake.

[0136] Optionally, the statistical information of the data partition includes the field cardinality of the data partition, and the global statistical information includes the field cardinality of the data lake.

[0137] The global statistical information determination module 14 is configured to perform statistics on the characteristic values ​​of the different data partitions stored in the first data structure to obtain characteristic values ​​of the data lake stored in the second data structure; convert the storage format of the characteristic values ​​of the data lake from the second data structure to the first data structure; and determine the field cardinality of the data lake based on the characteristic values ​​of the data lake stored in the first data structure.

[0138] Optionally, the first data structure includes a one-dimensional array, the second data structure includes a two-dimensional array, and the length of the one-dimensional array is the same as the number of rows of the two-dimensional array.

[0139] The global statistical information determination module 14 is configured to generate a two-dimensional array corresponding to the data lake based on the one-dimensional arrays corresponding to different data partitions in the data lake, wherein the target elements in the second array are the number of target values ​​in the one-dimensional array, the target elements correspond to the target number of rows and the target number of columns of the second array, the position of the target value in the one-dimensional array is the same as the target number of rows, and the target value is the same as the target number of columns.

[0140] Optionally, the apparatus further includes: a local statistical information determination module 15, configured to perform hash calculation on data in any data partition in the data lake;

[0141] Truncating the hash calculation result into a first part and a second part serving as an index, wherein the number of columns of the two-dimensional array is the same as the number of digits of the first part of the hash calculation result;

[0142] The maximum number of leading zeros in the first part of the hash calculation result is written into the position pointed to by the second part of the hash calculation result in the one-dimensional array to obtain a one-dimensional array corresponding to any data partition.

[0143] Optionally, the global statistical information determination module 14 is configured to sequentially traverse different rows in the two-dimensional array in a preset direction to obtain the first non-zero data in the different rows;

[0144] According to the order in which the non-zero data corresponding to different rows in the two-dimensional array are obtained, the number of columns of the non-zero data corresponding to the different rows in the two-dimensional array are sequentially written into the one-dimensional array corresponding to the data lake.

[0145] Optionally, the apparatus further includes: an updating module 16, configured to update a two-dimensional array corresponding to the data lake in response to additions and deletions of data partitions in the data lake.

[0146] Optionally, the updating module 16 is configured to, in response to the addition of a new data partition, obtain a one-dimensional array corresponding to the newly added data partition and a two-dimensional array corresponding to the data lake;

[0147] Traversing the one-dimensional array corresponding to the newly added data partition;

[0148] Increase the elements corresponding to the first reference row number and the first reference column number in the two-dimensional array corresponding to the data lake by one, wherein the first reference column number is the same as the first element in the one-dimensional array corresponding to the newly added data partition currently traversed, and the first reference row number and the first element have the same position in the one-dimensional array corresponding to the newly added data partition.

[0149] Optionally, the updating module 16 is configured to, in response to deletion of a data partition, obtain a one-dimensional array corresponding to the deleted data partition and a two-dimensional array corresponding to the data lake;

[0150] Traversing the one-dimensional array corresponding to the deleted data partition;

[0151] Subtract one from the elements corresponding to the second reference row number and the second reference column number in the two-dimensional array corresponding to the data lake, where the second reference column number is the same as the second element in the one-dimensional array corresponding to the currently traversed deleted data partition, and the second reference row number and the second element have the same position in the one-dimensional array corresponding to the deleted data partition.

[0152] Figure 6 The device shown can perform Figures 1 to 4 For the method of the embodiment shown in FIG. 1 , reference may be made to the description of the part not described in detail in the embodiment. Figures 1 to 4 The implementation process and technical effects of this technical solution can be found in Figures 1 to 4 The description in the illustrated embodiment will not be repeated here.

[0153] In a possible design, the data query method provided in the above embodiments can be applied in an electronic device, such as

[0154] Figure 7 As shown, the electronic device may include: a processor 31 and a memory 32. The memory 32 is used to store the data that supports the electronic device to execute the above Figures 1 to 4 The program of the data query method provided in the illustrated embodiment, the processor 31 is configured to execute the program stored in the memory 32 .

[0155] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the first processor 31, the following steps can be implemented:

[0156] Responding to a query request from the data lake, reading global statistical information of the data lake;

[0157] Determining a query strategy corresponding to the query request according to the global statistical information;

[0158] Respond to the query request according to the query strategy.

[0159] Optionally, the processor 31 is further configured to execute the aforementioned Figures 1 to 4 All or part of the steps in the illustrated embodiments.

[0160] The structure of the electronic device may further include a communication interface 33 for the electronic device to communicate with other devices or communication systems.

[0161] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by the above electronic device, which includes instructions for executing the above Figures 1 to 4 The procedures involved in the data query method shown.

[0162] In addition, an embodiment of the present invention provides a computer program product. The computer program product includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is enabled to implement the above Figures 1 to 4 The steps or functions of the data query method shown.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data query method, characterized in that: include: Responding to a query request from the data lake, reading global statistical information of the data lake; Determining a query strategy corresponding to the query request according to the global statistical information; Respond to the query request according to the query strategy.

2. The method according to claim 1, characterized in that The method is applied to a query engine, the global statistical information of the data lake is stored in a database shared by multiple data lakes, or stored in a cache of a query device, and the query engine is deployed in the query device.

3. The method according to claim 1, characterized in that The method further comprises: Determine global statistical information of the data lake based on statistical information of different data partitions in the data lake.

4. The method according to claim 3, characterized in that The statistical information of the data partition includes the field cardinality of the data partition, and the global statistical information includes the field cardinality of the data lake; Determining global statistical information of the data lake based on statistical information of different data partitions in the data lake includes: Performing statistics on the characteristic values ​​of the different data partitions stored in the first data structure to obtain characteristic values ​​of the data lake stored in the form of a second data structure; Converting the storage format of the characteristic values ​​of the data lake from the second data structure to the first data structure; The field cardinality of the data lake is determined according to the characteristic values ​​of the data lake stored in the first data structure.

5. The method according to claim 4, characterized in that The first data structure includes a one-dimensional array, the second data structure includes a two-dimensional array, and the length of the one-dimensional array is the same as the number of rows of the two-dimensional array; The performing statistics on the characteristic values ​​of the different data partitions stored in the first data structure to obtain the characteristic values ​​of the data lake stored in the second data structure includes: A two-dimensional array corresponding to the data lake is generated based on the one-dimensional arrays corresponding to different data partitions in the data lake, wherein the target elements in the second array are the number of target values ​​in the one-dimensional array, the target elements correspond to the target number of rows and the target number of columns of the second array, the position of the target value in the one-dimensional array is the same as the target number of rows, and the target value is the same as the target number of columns.

6. The method according to claim 5, characterized in that The method further comprises: Performing hash calculation on data in any data partition in the data lake; Truncating the hash calculation result into a first part and a second part serving as an index, wherein the number of columns of the two-dimensional array is the same as the number of digits of the first part of the hash calculation result; The maximum number of leading zeros in the first part of the hash calculation result is written into the position pointed to by the second part of the hash calculation result in the one-dimensional array to obtain a one-dimensional array corresponding to any data partition.

7. The method according to claim 5, characterized in that The converting the storage format of the characteristic values ​​of the data lake from the second data structure to the first data structure includes: Traversing different rows in the two-dimensional array in sequence according to a preset direction to obtain the first non-zero data in the different rows; According to the order in which the non-zero data corresponding to different rows in the two-dimensional array are obtained, the number of columns of the non-zero data corresponding to the different rows in the two-dimensional array are sequentially written into the one-dimensional array corresponding to the data lake.

8. The method according to claim 5, characterized in that The method further comprises: In response to the addition or deletion of data partitions in the data lake, the two-dimensional array corresponding to the data lake is updated.

9. The method according to claim 8, characterized in that In response to the addition or deletion of data partitions in the data lake, updating the two-dimensional array corresponding to the data lake includes: In response to the addition of a data partition, obtaining a one-dimensional array corresponding to the newly added data partition and a two-dimensional array corresponding to the data lake; Traversing the one-dimensional array corresponding to the newly added data partition; Increase the elements corresponding to the first reference row number and the first reference column number in the two-dimensional array corresponding to the data lake by one, wherein the first reference column number is the same as the first element in the one-dimensional array corresponding to the newly added data partition currently traversed, and the first reference row number and the first element have the same position in the one-dimensional array corresponding to the newly added data partition.

10. The method according to claim 8, characterized in that In response to the addition or deletion of data partitions in the data lake, updating the two-dimensional array corresponding to the data lake includes: In response to deletion of a data partition, obtaining a one-dimensional array corresponding to the deleted data partition and a two-dimensional array corresponding to the data lake; Traversing the one-dimensional array corresponding to the deleted data partition; Subtract one from the elements corresponding to the second reference row number and the second reference column number in the two-dimensional array corresponding to the data lake, where the second reference column number is the same as the second element in the one-dimensional array corresponding to the currently traversed deleted data partition, and the second reference row number and the second element have the same position in the one-dimensional array corresponding to the deleted data partition.

11. A query engine, characterized in that: include: processing components and query optimization components; The processing component is configured to read global statistical information of the data lake in response to a query request of the data lake; Responding to the query request according to the query strategy determined by the query optimization component; The query optimization component is configured to determine a query strategy corresponding to the query request according to the global statistical information.

12. The engine according to claim 10, characterized in that The statistical information of the data partition includes the field cardinality of the data partition, and the global statistical information includes the field cardinality of the data lake, where the field cardinality is stored in the form of a first data structure; The information processing component is configured to count the field cardinalities of the different data partitions stored in the first data structure to obtain the field cardinalities of the data lake stored in the second data structure; The storage format of the field cardinality of the data lake is converted from the second data structure to the first data structure.

13. An electronic device, characterized in that: include: A memory and a computing system; wherein the memory stores executable code, and when the executable code is executed by the computing system, the computing system executes the data query method according to any one of claims 1 to 10.

14. A non-transitory machine-readable storage medium, characterized in that The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a computing system of an electronic device, the computing system is caused to execute the data query method according to any one of claims 1 to 10.

15. A computer program product, characterized in that The method comprises a computer program or instructions, which, when executed by a processor, enables the processor to implement the steps of the data query method according to any one of claims 1 to 10.

Images