Data query method and device, equipment cluster, program product and storage medium

Abstract

A data query method and device, equipment cluster, program product and storage medium are disclosed, and relate to the technical field of data compression. The encoded values of part of original data in a database are divided into multiple partitions, and based on the partition dictionaries of the multiple partitions, a target partition containing the original data to be queried is first determined, and querying only in the target partition can reduce the amount of data to be queried; then, the encoded value corresponding to the original data to be queried is determined based on the target partition dictionary, and a query operation is directly performed on the encoded value to be queried in the target partition to obtain a processing result; compared with the query operation on the decoded string in the file-level dictionary encoding, the data amount of the encoded value is smaller than that of the string, and the query effect of the query operation on the encoded value to be queried is higher. Based on the target partition dictionary, the processing result only needs to be decoded once to obtain a query result, thereby improving the query efficiency when querying data.

Description

Technical Field

[0001] This application relates to the field of data compression technology, and in particular to data query methods, apparatus, equipment clusters, program products and storage media. Background Technology

[0002] Dictionary encoding refers to replacing repeated original data in a file with corresponding encoded values ​​based on a dictionary. Since the length of the original data is longer than the length of the encoded values, data compression can be achieved after data encoding. The dictionary contains each piece of original data in the file and its corresponding encoded value.

[0003] File-level dictionary encoding is applied to a single file. Each file has its own independent file dictionary, and each file dictionary corresponds to different encoded values ​​for the same original data. Therefore, if a cross-file query is required, it is necessary to decode multiple times based on multiple independent dictionaries and then perform the query operation on the decoded original data, resulting in low query efficiency.

[0004] Therefore, improving query efficiency has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a data query method, apparatus, equipment cluster, program product, and storage medium that can improve query efficiency.

[0006] Firstly, a data query method is provided. This method is used to query a database, which includes multiple partitions. Each partition contains encoded values ​​corresponding to a portion of the original data in the database. The method includes: obtaining a query request, which contains the original data to be queried and a query operation; determining the target partition containing the original data to be queried based on the original data to be queried and the partition dictionaries of the multiple partitions; determining the queried encoded value corresponding to the original data to be queried based on the target partition dictionary of the target partition; performing a query operation on the queried encoded value in the target partition to obtain a processing result; and decoding the processing result based on the target partition dictionary to obtain the query result.

[0007] Based on the aforementioned data query method, the encoded values ​​of the original data in the database are divided into multiple partitions, each containing a portion of the database's encoded values. First, based on the partition dictionary of multiple partitions, the target partition containing the original data to be queried is determined. Querying only in the target partition reduces the amount of data to be queried. Then, based on the target partition dictionary, the corresponding encoded value to be queried for the original data is determined, and the query operation is directly performed on the encoded value in the target partition to obtain the processing result. Compared to querying the decoded string in file-level dictionary encoding, since the data volume of the encoded value is smaller than that of the string, this application achieves higher query efficiency by directly querying the encoded value. Since the processing result based on the target partition dictionary only requires one decoding operation to obtain the query result, query efficiency is improved during data querying.

[0008] One possible implementation is that the query request can include a filter query request containing a shard-based dictionary (SDICT) field, which is used to construct the partition dictionary. In this way, by including the partition dictionary field in the filter query request, queries can be performed on a partition-by-partition basis, improving query efficiency compared to scanning the entire database.

[0009] Based on the query code value and each file in the target partition, the files containing the query code value are identified. Within these files, a filtering query operation is performed on the query code value to obtain the processing result. By identifying the query files based on each file in the target partition, file-level filtering during the query is achieved. Querying only within the target files reduces the amount of data to be queried, further improving the efficiency of the filtering query.

[0010] As one possible implementation, the query request includes an aggregate (agg) query request, which contains dictionary-encoded fields. The aggregate function from the aggregate query request is retrieved; aggregation is then performed on the target partition according to the aggregate function and the coded value to be queried, yielding the processed result. Thus, in the target partition, an aggregate query is first performed based on the coded value to obtain the processed result, and then the coded value is decoded to obtain the decoded result. Compared to the file-level dictionary encoding method, which performs aggregation queries based on the decoded string, the coded value to be queried has a smaller data volume than the string, further improving the query efficiency of the aggregation query. Compared to the global dictionary encoding method, which performs aggregation queries globally within the database, this application only needs to perform aggregation queries on the target partition, reducing the amount of data to be queried and further improving the query efficiency of the aggregation query.

[0011] As one possible implementation, the query request includes a groupby query request, which contains a dictionary-encoded field. The target partition is grouped according to the queried encoded value to obtain the processing result. Thus, in the target partition, a group query is first performed based on the queried encoded value to obtain the processing result, and then the queried encoded value is decoded to obtain the decoded result. Compared to the file-level dictionary encoding method, which performs group queries based on the decoded string, the queried encoded value has a smaller data volume than the string, resulting in higher efficiency for group queries. Compared to the global dictionary encoding method, which performs group queries globally within the database, this application only needs to perform group queries in the target partition, reducing the amount of data to be queried and further improving the query efficiency of group queries.

[0012] As one possible implementation, the target partition comprises multiple shards, each containing a portion of the target partition's encoded values. Each shard is partitioned based on the file storage time of the files within the target partition. After determining the target partition, the target shard containing the original data to be queried is determined based on the original data to be queried and the shard dictionary corresponding to each shard. Each shard dictionary indicates a partial mapping relationship between the target partition dictionary and contains the mapping relationship between the partial encoded values ​​of each shard and the original data. Based on the target shard dictionary of the target shard, the queried encoded value corresponding to the original data to be queried is determined. A query operation is performed on the queried encoded value in the target shard to obtain the processing result. Since each shard contains a portion of the target partition's encoded values, the target shards can be further refined after the target partition is determined, achieving shard-level filtering during queries. Querying only within the target shard reduces the amount of data to be queried, further improving query efficiency. The processing result is decoded based on the target shard dictionary to obtain the query result. Since each partition dictionary is a part of the target partition dictionary, the amount of data during decoding based on the target shard dictionary is further reduced, further improving query efficiency.

[0013] Optionally, the query request includes a filtering query request, which contains a dictionary-encoded field. Based on the query-to-be-queried code value and each file in the target shard, the file containing the query-to-be-queried code value is determined; in the file to be queried, the query-to-be-queried code value is filtered, and the processing result is obtained.

[0014] Optionally, the query request includes an aggregation query request, which contains dictionary-encoded fields. The aggregation function from the aggregation query request is retrieved, and the target shard is aggregated according to the aggregation function and the coded value to be queried to obtain the processing result.

[0015] Optionally, the query request includes a grouped query request, which contains a dictionary-encoded field. The target shard is then grouped according to the queried encoded value to obtain the processing result.

[0016] All three of the above-mentioned optional implementation methods perform queries after the target shard is determined. Compared with the previous scheme of performing queries after the target partition is determined, since the target shard only includes a portion of the target partition's encoded values, the amount of data in the target shard is further reduced, thereby improving the query efficiency for filtering queries, aggregation queries, and grouping queries.

[0017] As one possible implementation, when the storage duration of the target shard in the target partition exceeds a preset duration threshold, the target shard dictionary is deleted from the target partition dictionary. This achieves aging management of data in the partition dictionary, thus avoiding the problem of the partition dictionary gradually expanding over time, and preventing the gradual decrease in storage and query efficiency as the number of coded values ​​in the partition increases.

[0018] One possible implementation involves obtaining the file to be written; determining the target original data not included in the partition dictionaries of multiple partitions based on the original data in the file; and adding the target original data and its corresponding target encoding value to the partition dictionaries of multiple partitions based on the dictionary encoding field, thus obtaining the updated partition dictionaries of multiple partitions. In this way, when data is updated in the database, only the partition dictionaries of multiple partitions need to be updated, reducing the update cost compared to file-level dictionary encoding methods that require creating a new file dictionary for each new file. Compared to global-level dictionary encoding methods, which require the central node to communicate with each data node to ensure that the file dictionaries of all files are updated synchronously after updating the global dictionary, this application allows files in each partition to share a single partition dictionary, requiring only the updates of the partition dictionaries of multiple partitions, thereby avoiding the high communication overhead caused by synchronous updates in global-level dictionary encoding methods.

[0019] In a second aspect, a data query apparatus is provided, the apparatus comprising modules for performing the data query method in the first aspect or any possible implementation thereof.

[0020] The data query device described in the second aspect can be a terminal device or a network device, or it can be a chip (system), network card or other component or assembly that can be set in a terminal device or a network device, or it can be a device that includes a terminal device or a network device. This application does not limit it in this regard.

[0021] Furthermore, the technical effects of the data query device described in the second aspect can be referred to the technical effects of the data query method described in the first aspect, and will not be repeated here.

[0022] Thirdly, a computing device is provided, the computing device including a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are recorded and executed by the processor to enable the computing device to perform the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

[0023] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and memory;

[0024] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any possible implementation of the first aspect.

[0025] Fifthly, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, causing the computing device to perform operational steps of the method as described in any possible implementation of the first aspect.

[0026] In a sixth aspect, a chip system is provided. The chip system includes a memory and at least one processor. The memory stores a set of computer instructions, which, when executed by the processor, perform the operational steps of the method described in any possible implementation of the first aspect.

[0027] In a seventh aspect, a computer program product is provided that, when run on a computing device, causes the computing device to perform the operational steps of the method as described in any possible implementation of the first aspect.

[0028] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of a file-level dictionary encoding.

[0030] Figure 2 This is a schematic diagram of a global-level dictionary encoding.

[0031] Figure 3 A schematic diagram of the architecture of a cloud platform 300 provided for an embodiment of this application;

[0032] Figure 4 A schematic diagram of the layered structure of a cloud platform 300 provided in an embodiment of this application;

[0033] Figure 5 A flowchart illustrating a data query method provided in an embodiment of this application;

[0034] Figure 6 A schematic diagram of a partition already constructed in a database, provided as an embodiment of this application;

[0035] Figure 7 A flowchart illustrating a data query method provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram illustrating a sharding method for constructing files in a database after performing two-level partitioning, as provided in an embodiment of this application.

[0037] Figure 9 A schematic diagram of a segmented dictionary encoding provided for an embodiment of this application;

[0038] Figure 10 A flowchart illustrating a data query method provided in an embodiment of this application;

[0039] Figure 11 A flowchart illustrating a partition dictionary construction process provided in this application embodiment;

[0040] Figure 12 A schematic diagram of a partition module provided for an embodiment of this application;

[0041] Figure 13 A flowchart illustrating a segmented dictionary construction process provided in this application embodiment;

[0042] Figure 14 This is a schematic diagram of the structure of a data query device 1400 provided in an embodiment of this application;

[0043] Figure 15 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0044] Figure 16 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0045] Figure 17 This is a schematic diagram of a network connection structure between computing devices provided in an embodiment of this application. Detailed Implementation

[0046] This application can be applied not only to existing data storage systems and data compression scenarios, but also to future data storage systems and data compression scenarios. This application can be executed by a device or cluster of devices with a data storage system, such as terminal devices, network devices, computing devices, storage devices, and cloud platforms composed of computing devices. The terminology used in the implementation section of this application is only used to explain specific embodiments of this application and is not intended to limit this application.

[0047] To facilitate understanding, the relevant terms involved in the embodiments of this application will be introduced below.

[0048] Dictionary encoding is a data compression technique.

[0049] In dictionary encoding, the dictionary stores all unique values ​​in a file and their corresponding encoded values. Duplicate values ​​in the file are replaced with their corresponding encoded values. When there is a large amount of repetitive original data in a file, dictionary encoding can significantly reduce the storage resources consumed by file storage. A unique value refers to each piece of original data that appears at least once in the file.

[0050] To make the technical issues easier to understand, the existing data query methods are described below.

[0051] When the data node's storage system uses a columnar storage format (Apache Parquet) or a columnar storage engine (Apache Optimized Row Columnar, Apache ORC), the data node uses dictionary encoding to compress files.

[0052] Each data node creates a dictionary for each column in each file. When a data node receives raw data written to a column, it looks up the unique value in the class and assigns an encoding value to the unique value. When a data node receives a data read request, it converts the encoding value in the column back into the raw data and then outputs it.

[0053] Dictionary encoding has the advantage of compressing data without knowing its contents, making it suitable for real-time data compression scenarios. However, when there is not a large amount of repetitive original data in the file, the compression effect of dictionary encoding is not significant, and building and maintaining the dictionary increases the complexity of the data compression and decompression process.

[0054] File-level dictionary encoding is applied to a single file and is suitable for columns of raw data that are repeated extensively within a single file. Examples of file-level dictionary encoding include columnar database management systems like ClickHouse for online analytical processing (OLAP) and open-source databases like InfluxDDB for storing and analyzing time-series data.

[0055] Figure 1 This is a schematic diagram of a file-level dictionary encoding. For example... Figure 1 As shown, Figure 1The file dictionary of data file 1 (datefile1) includes the string "efsdgere" and its corresponding encoding value 1, the string "ghnrtbhm" and its corresponding encoding value 2, the string "cvdtgder" and its corresponding encoding value 3, the string "sfdvbmrt" and its corresponding encoding value 4, the string "tujgnhju" and its corresponding encoding value 5, and the string "dfgdoju" and its corresponding encoding value 6.

[0056] Figure 1 The reverse index of data file 1 includes the string "ghnrtbhm" and its corresponding encoded value 2, the string "tujgnhju" and its corresponding encoded value 5, the string "dfgdoju" and its corresponding encoded value 6, the string "sfdvbmrt" and its corresponding encoded value 4, the string "cvdtgder" and its corresponding encoded value 3, and the string "efsdgere" and its corresponding encoded value 1. The reverse index is used to convert the encoded values ​​back to the original strings when the data is read.

[0057] Figure 1 Each string in the original column positions of data file 1 is replaced with its corresponding encoded value.

[0058] Based on the above Figure 1 The relevant description states that file-level dictionary encoding is for columns containing a large number of repeated strings. A file dictionary is created based on the repeated strings in the column, and the repeated strings in the column are replaced based on the encoded values ​​in the file dictionary, thereby reducing the storage resources required for the column.

[0059] The advantage of file-level dictionary encoding is that it is relatively simple and easy to implement, but the data compression effect of file-level dictionary encoding is limited by the degree of data duplication in a single file.

[0060] Since each file has its own independent file dictionary, the same string in the file dictionaries of different files has different encoding values. When it is necessary to query across files in the database, it is necessary to decode multiple times based on the file dictionary corresponding to each file and then perform query operations on the decoded strings, which results in low query efficiency.

[0061] Furthermore, since each file requires its own file dictionary, the maintenance cost of the file dictionary increases when the data in the file is frequently updated. The file dictionaries of different files are not shared, and the same data needs to be stored repeatedly in different file dictionaries, resulting in wasted storage space.

[0062] Global dictionary encoding is applied to databases and is suitable for repetitive data throughout the database. For example, global dictionary encoding can be used in high-performance analytical data warehouses (like StarrokCS).

[0063] Global dictionary encoding is achieved by creating a global dictionary containing all repeating data, which is then shared and reused throughout the database. The advantage of global dictionary encoding is that it can share encoding across files, resulting in higher compression rates and query performance. Global dictionary encoding is suitable for values ​​that appear repeatedly in multiple files.

[0064] Figure 2 This is a schematic diagram of a global dictionary encoding.

[0065] like Figure 2 As shown, Figure 2 Each datanode in the system uses global-level dictionary encoding. Datanode 0 contains multiple files, each encoded using file-level dictionary encoding. When a file in datanode 0 changes, a meta-node update global is performed. During a global dictionary update, the file dictionary in datanode 0 must be updated synchronously.

[0066] As the amount of data in the database gradually increases over time, the global dictionary expands, which affects the query efficiency when using the global dictionary and requires a large amount of storage space.

[0067] Furthermore, updating the global dictionary requires ensuring that updates to all related files in the database are timely, leading to significant network overhead due to synchronous updates. Additionally, locking is necessary during global dictionary updates to prevent issues caused by the update from degrading write operations.

[0068] To address the low query efficiency issue of file-level dictionary encoding, this application provides a data query method. The method divides the encoded values ​​of a portion of the original data in the database into multiple partitions. Based on the partition dictionaries of these partitions, the target partition containing the original data to be queried is first determined. Querying only in the target partition reduces the amount of data to be queried. Then, based on the target partition dictionary, the corresponding encoded value to be queried for the original data is determined, and the query operation is directly performed on the encoded value in the target partition to obtain the processing result. Compared to querying the decoded string in file-level dictionary encoding, the query efficiency is higher because the encoded value has a smaller data volume than the string. Since the processing result based on the target partition dictionary only requires one decoding operation to obtain the query result, the query efficiency is improved during data querying.

[0069] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0070] Figure 3 This is a schematic diagram of the architecture of a cloud platform 300 provided in an embodiment of this application. Figure 3 As shown, the cloud platform 300 includes a computing server cluster 310, a storage server cluster 320, a management server cluster 330, a network device cluster 340, and a user terminal 350. The computing server cluster 310, storage server cluster 320, and management server cluster 330 communicate with the user terminal 350 through the network device cluster 340.

[0071] The computing server cluster 310 includes one or more computing servers ( Figure 3 The diagram shows two computing servers, namely computing server 311 and computing server 312, but is not limited to two computing servers.

[0072] Computing servers, such as servers and desktop computers, are computing resources within the cloud platform 300. They are used to generate and allocate computing resources based on user needs using virtualization technology. At the hardware level, computing servers are equipped with processors and memory. Figure 3 (Not shown in the diagram), the computing server's computing functions are implemented by the processor running programs in memory. The computing server can also read / write data on various storage servers in the storage server cluster 320 according to user needs.

[0073] Storage server cluster 320 includes one or more storage servers ( Figure 3 The image shows two storage servers, namely storage server 321 and storage server 322, but is not limited to two storage servers.

[0074] Storage servers, serving as storage resources within the cloud platform 300, include servers, desktop computers, storage array controllers, and hard drive enclosures. They provide logical disk storage, unstructured data storage, and integrated backup services for cloud virtual machines within the cloud platform 300. In terms of hardware, storage servers are equipped with network interface cards (NICs), processors, and memory. The processor in the storage server processes data from outside the server. The NIC controls access to the memory, such as address signals, data signals, and various command signals, enabling the storage server to provide memory as a storage resource to users. Memory stores data and may include RAM and / or hard drives. RAM refers to internal storage that directly exchanges data with the processor; it can quickly read and write data at any time, serving as temporary data storage for the operating system or other running programs. Unlike RAM, hard drives are slower to read and write data and are typically used for persistent data storage.

[0075] Management server cluster 330 includes one or more management servers ( Figure 3 The diagram shows two management servers, namely management server 331 and management server 332, but is not limited to two management servers.

[0076] The management server is used to manage all computing services, shared storage, and network of the entire cloud platform 300, and also provides users or administrators with an API to manage the entire node.

[0077] Network device cluster 340 includes one or more switches and routers, such as Figure 3 As shown, in this embodiment, the network device cluster 340 includes a router 341, a switch 342, a switch 343, a switch 344, and a switch 345. The user terminal 350 is connected to the router 341 via the Internet. The router 341 is then connected to the switch 343 via the switch 342. The switch 343 is connected to each computing server in the computing server cluster 310. The switch 344 is connected to each computing server in the computing server cluster 310 and each storage server in the storage server cluster 320. The switch 345 is connected to each computing server in the computing server cluster 310, each storage server in the storage server cluster 320, and each management server in the management server cluster 330.

[0078] Optionally, the number and type of switches included in the network device cluster 340 can be adjusted according to the needs of the cloud platform 300. Switches 342, 343, 344, and 345 can be switches with different functions. For example, switch 342 is the core switch, while switches 343, 344, and 345 are used to manage specific network segments. For instance, switch 342 can be the core switch, switch 343 can be an internal / external switching network segment switch, switch 344 can be a storage network segment switch, and switch 345 can be a management network segment switch.

[0079] User terminal 350 includes one or more user terminals ( Figure 3 The diagram shows two user terminals, namely user terminal 351 and user terminal 352, but is not limited to two user terminals. The user terminals contain the interfaces and applications required for accessing the cloud platform 300.

[0080] It is worth noting that, Figure 3 This is merely an illustration and should not be construed as limiting the scope of this application. The cloud platform 300 may also include other devices. Figure 3 It is not shown in the middle.

[0081] In such Figure 3 Based on the equipment of the cloud platform 300 shown, the cloud platform 300 implements the functions of service nodes based on IaaS, PSSS, and SaaS, and provides services (e.g., computing services, storage services, and network services) to users through service nodes. These service nodes can be cloud nodes (e.g., data nodes) virtualized using the resources (e.g., computing resources and storage resources) of the cloud platform 300.

[0082] Next, combine Figure 4 The layered structure of the cloud platform 300 is explained.

[0083] Figure 4 This is a schematic diagram of the layered structure of a cloud platform 300 provided in an embodiment of this application. For example... Figure 4 As shown, the Infrastructure as a Service platform 410 is used to virtualize all infrastructure resources in the cloud platform 300, providing virtual resources (e.g., computing resources, network resources, and storage resources) to users in a software-defined manner. Infrastructure resources refer to the resources provided by the computing server cluster 310, storage server cluster 320, management server cluster 330, and / or network device cluster 340 in the cloud platform 300.

[0084] The Platform as a Service (PaaS) platform 420 provides the runtime environment and application support functions for the cloud platform 300, allowing users to request compute units within their quotas instead of virtual resources to run their services. Optionally, the compute units can be containers, and the cloud platform 300 deploys and runs user code by scheduling containers. It should be noted that the number of containers in the Platform as a Service (PaaS) platform 420 can be one or more. Figure 4 The example uses only two containers.

[0085] As one possible implementation, the cloud platform 300 can inject one or more components into a container to deploy and run the code. Optionally, the resources (computing or storage resources) used by multiple components in the same container can belong to the same hardware device (e.g., a compute server, storage server, or management server) or different hardware devices within the cloud platform 300.

[0086] The components may include database components, message queue components, orchestration components, logging components, virtualization components, business database components, configuration database components, and permission management components.

[0087] Software as a Service (SaaS) application 430 is used to provide services to users by composing user-deployed applications as API responses, based on Infrastructure as a Service (IaaS) platform 410 and Platform as a Service (PAS) platform 420. The application of SaaS application 430 and the container can communicate with each other through a web server.

[0088] It is worth noting that, Figure 4 This is merely an illustration and should not be construed as limiting the scope of this application. The layered structure of the cloud platform 300 may also include other modules. Figure 4 It is not shown in the middle.

[0089] The data query method provided in this application embodiment can be executed by a service node (e.g., a data node) provided by the cloud platform 300, and can be applied to the storage and querying of high-frequency low-cardinality strings in online transaction processing (OLTP) and online analytical processing (OLAP) scenarios of data nodes. Here, high-frequency low-cardinality strings refer to strings of low cardinality type that appear frequently. The following will combine... Figure 5 The data query method provided in the embodiments of this application will be described.

[0090] Figure 5 This is a flowchart illustrating a data query method provided in an embodiment of this application. The steps in this data query method can be... Figure 4 The data nodes shown are executed, such as Figure 5 As shown, the data query method may include steps 501 to 505.

[0091] Step 501: Data nodes obtain query requests.

[0092] The data node receives a query request to query the database of cloud platform 300. The query request includes the original data to be queried and the query operation. The database contains encoded values ​​of the original data. The database consists of multiple partitions, and each partition contains a portion of the encoded values ​​of the database.

[0093] The query request may include at least one of the following: a filtering query request, an aggregation query request, and a grouping query request.

[0094] The original data can be a string, low cardinality data, etc., and the coded value to be queried can be a positive integer value.

[0095] Partitioning is a database table design method used to divide the data in a table into smaller, more manageable parts. Each partition can store a subset of data based on specific rules (such as date ranges, numeric ranges, or list values). Because partitioned database tables allow scanning only relevant partitions instead of the entire table, query performance can be improved. Furthermore, partitioning can simplify data management tasks such as data backup and recovery.

[0096] For example, when the original data is a string, the data node divides the files in the cloud platform 300 database into multiple partitions based on the dictionary encoding field and multiple string groups. For each partition, the data node constructs a partition dictionary by establishing a mapping relationship between the strings and their encoding values. The specific steps for the data node to construct the partition dictionary are detailed below. Figure 11 The relevant descriptions are not repeated here in the embodiments of this application.

[0097] The target partition includes at least one target file, and the at least one target file includes strings from the target string group corresponding to the target partition. The target string group is one of multiple string groups obtained by evenly dividing the unique values ​​in multiple files in the database. The multiple files include at least one target file, and the unique values ​​are used to indicate multiple strings that appear at least once in the multiple files.

[0098] The following example describes the partitions that have been built in a data node. Figure 6 This is a schematic diagram of a partition already constructed in a database, provided as an embodiment of this application.

[0099] like Figure 6As shown, data nodes divide the database into multiple partitions vertically based on the shard key, and maintain the mapping relationship of the partition dictionary in the partition, that is, the mapping relationship between strings and encoded values ​​and the mapping relationship between encoded values ​​and strings.

[0100] In the partition dictionary, the encoded values ​​are incremented in the order in which they appear. For example, the encoded values ​​in the partition dictionary start from 1, the data type is a 19-bit unsigned integer array (unit16), the storage size of the encoded value is 2 bytes, and the data range of the encoded value is 0 to 65535.

[0101] A partition can contain multiple files, and all files within the partition share the partition dictionary. This means that for the same string, there is exactly one encoding value corresponding to that string. Therefore, compared to the storage waste caused by duplicate storage in file-level dictionary encoding and the high maintenance cost of file dictionaries, partition dictionary encoding avoids the storage waste caused by duplicate storage. Furthermore, since multiple files in a partition only need to maintain one partition dictionary, the maintenance cost of the partition dictionary is lower.

[0102] Furthermore, compared to the problem of high network transmission overhead caused by cross-node communication for synchronous updates in global dictionary encoding, the shared partition dictionary of different files in the same partition in this embodiment does not require cross-node communication, thus avoiding the problem of high network transmission overhead.

[0103] In this method, data nodes use hash partitioning to divide multiple files in the database into multiple partitions corresponding to multiple string groups. Hash partitioning can be any of the following: node modulo hashing, consistent hashing, virtual slot hashing, etc.

[0104] As one possible implementation, the data node obtains the query request, determines that the query request contains a dictionary-encoded field, and then identifies the original data to be queried in the query request. The dictionary-encoded field is used to construct multiple partition dictionaries.

[0105] Step 502: The data node determines the target partition containing the original data to be queried based on the original data to be queried and the partition dictionary of multiple partitions.

[0106] Each partition dictionary contains a mapping between the encoded values ​​in each partition and the original data. This mapping is bidirectional, meaning that the partition dictionary includes both a mapping from the original data to the encoded values ​​and a mapping from the encoded values ​​to the original data.

[0107] The data node obtains partition dictionaries for multiple partitions from multiple partitions, and determines the target partition containing the original data to be queried based on the original data contained in the multiple partition dictionaries.

[0108] Step 503: The data node determines the query code value corresponding to the original data to be queried based on the target partition dictionary of the target partition.

[0109] The data node determines the coded value pointed to by the original data to be queried as the coded value based on the mapping relationship between the original data and the coded value in the target partition dictionary.

[0110] Step 504: The data node performs a query operation on the queried code value in the target partition to obtain the processing result.

[0111] As one possible implementation, data nodes perform query operations on the queried encoded value in the target partition, and the operation can be performed according to the specific query request. This application provides the following specific implementations of data nodes performing query operations on the queried encoded value under different query requests, but these do not constitute specific limitations.

[0112] In the first optional implementation, the query request may include a filter query request, which includes a dictionary-encoded field. The data node determines the file containing the query code value based on the code value and each file in the target partition. The data node then performs a filter query operation on the query code value in the file to be queried, obtaining the processing result.

[0113] In the second optional implementation, the query request may include an aggregation query request, which contains a dictionary-encoded field. The data node retrieves the aggregation function from the aggregation query request, performs aggregation processing on the target partition according to the aggregation function and the coded value to be queried, and obtains the processing result.

[0114] Aggregate functions are functions that calculate a set of values ​​and return a single value. For example, aggregate functions can be any of the following: summation function, average function, maximum value function, minimum value function, statistical function, etc.

[0115] In the third optional implementation, the query request may include a grouped query request, which contains a dictionary-encoded field. Data nodes group the data in the target partition according to the queried encoded value to obtain the processing result.

[0116] For example, data nodes can perform hash grouping on the target partition according to the code to be queried, resulting in each grouped file group.

[0117] Hash grouping refers to grouping multiple files using a hash algorithm.

[0118] As one possible implementation, data nodes can also broadcast query requests to each partition, determine whether the partition dictionary contains the queried code value in each partition, and perform a query operation on the queried code value corresponding to the original data to be queried in the target partition to obtain the processing result of the target partition.

[0119] Step 505: The data node decodes the processing result based on the target partition dictionary to obtain the query result.

[0120] The data nodes decode the processing results based on the mapping relationship between the encoded values ​​in the target partition and the original data to obtain the query results.

[0121] Based on the above Figure 5 In steps 501 to 505, the data node divides the encoded values ​​of the original data in the database into multiple partitions. Based on the partition dictionary of these partitions, the data node first determines the target partition containing the original data to be queried. Querying only in the target partition reduces the amount of data to be queried. Then, the data node determines the corresponding encoded value of the original data to be queried based on the target partition dictionary and directly performs the query operation on the encoded value in the target partition to obtain the processing result. Compared to querying the decoded string in file-level dictionary encoding, the query efficiency is higher because the encoded value has a smaller data volume than the string. Finally, the data node only needs to decode the processing result once based on the target partition dictionary to obtain the query result, thus improving query efficiency during data retrieval.

[0122] The preceding text described the complete data query process when partitioning files in a database. This application does not limit the granularity of data partitioning in the database. For example, files within a partition can be further divided into fragments. The target partition includes multiple fragments, each fragment containing a portion of the encoded values ​​from the target partition. Each fragment is obtained based on the file storage time of the files in the target partition, and the file storage time of the files in each fragment falls within the corresponding time range of that fragment. The following section will combine... Figure 5 The data query process when a file in a database is partitioned into two levels is described.

[0123] Figure 7 This is a flowchart illustrating a data query method provided in an embodiment of this application.

[0124] like Figure 7 As shown, the data query method includes steps 701 to 706.

[0125] Step 701: Data nodes obtain query requests.

[0126] Step 701 and above Figure 5 Step 501 is similar to that in the previous embodiment, and will not be described in detail here.

[0127] Step 702: The data node determines the target partition containing the original data to be queried based on the original data to be queried and the partition dictionary of multiple partitions.

[0128] Step 702 and above Figure 5 Step 502 is similar to that in the previous embodiment, and will not be described in detail here.

[0129] Step 703: The data node determines the target shard containing the original data to be queried based on the original data to be queried and the shard dictionary corresponding to each shard in the target partition.

[0130] As one possible implementation, the data node obtains the shard dictionary for each shard from the target partition, and determines the target shard containing the original data to be queried based on the original data contained in each shard dictionary.

[0131] Each segment dictionary contains partial mappings in the target partition dictionary corresponding to the partial encoded values ​​of each segment.

[0132] Sharding is a concept in distributed databases. It involves dividing data into multiple fragments and distributing these fragments across different servers or database instances. The purpose of sharding is to achieve load balancing and improve scalability, allowing database systems to handle more data and query requests by adding more servers.

[0133] For example, when the original data is a string, the data node divides the database files in the cloud platform 300 into multiple partitions based on the dictionary encoding field and multiple string groups. Then, based on the time range corresponding to each shard within a partition and the storage time of the files within that partition, the data node further divides the files within that partition into multiple shards. For each shard, the data node determines its shard dictionary based on the mapping relationship between the encoded values ​​in the partition dictionary and the shard values. The specific steps for the data node to construct the shard dictionary can be found later in this text. Figure 13 The embodiments of this application will not be repeated here.

[0134] The following example describes the shards that have been built in the data node. Figure 8 This is a schematic diagram illustrating a sharding method for constructing a database file after performing a two-level partitioning, as provided in an embodiment of this application.

[0135] like Figure 8As shown, the database is partitioned along two dimensions: time and shard key. Based on the shard key, it is vertically divided into three partitions: partition 0, partition 1, and partition 2. Based on time, it is horizontally divided into three shard groups: shard group 0, shard group 1, and shard group 2. The smallest unit after partitioning is a shard. Partition 0 is divided into three shards located in shard groups 0, 1, and 2. Partition 1 is divided into three shards located in shard groups 0, 1, and 2. Partition 2 is divided into three shards located in shard groups 0, 1, and 2.

[0136] The two-level partitioning mechanism can expand horizontally based on the number of unique values ​​and manage vertically based on time ranges. Files in the partitions have time attributes and can effectively balance the distribution of data, improving the query performance of time ranges.

[0137] A shard can contain multiple files. All files in a shard share the shard dictionary, meaning that for the same string, there is one and only one encoding value corresponding to that string.

[0138] Figure 9 This is a schematic diagram of a segmented dictionary encoding provided in an embodiment of this application, as shown below. Figure 9 As shown, Figure 9 The shard data file in the middle shard contains data file 1 and data file 2.

[0139] The shard dictionary encoding corresponding to each shard includes key-to-value and value-to-key encoding.

[0140] Figure 9 The key-to-value pairs in the middle shard include the string "efsdgere" and its corresponding encoding value 1, the string "ghnrtbhm" and its corresponding encoding value 2, the string "cvdtgder" and its corresponding encoding value 3, the string "sfdvbmrt" and its corresponding encoding value 4, the string "tujgnhju" and its corresponding encoding value 5, and the string "dfgdoju" and its corresponding encoding value 6.

[0141] Figure 9The values ​​to keys of the middle shards include encoded value 1 and the corresponding string "efsdgere", encoded value 2 and the corresponding string "ghnrtbhm", encoded value 3 and the corresponding string "cvdtgder", encoded value 4 and the corresponding string "sfdvbmrt", encoded value 5 and the corresponding string "tujgnhju", and encoded value 6 and the corresponding string "dfgdoju".

[0142] Figure 9 In the middle shard, each original string in the original column position of data file 1 and data file 2 is replaced with the encoded value corresponding to the string in the shard dictionary.

[0143] Step 704: The data node determines the query code value corresponding to the original data to be queried based on the target shard dictionary of the target shard.

[0144] The data node determines the coded value pointed to by the original data to be queried as the coded value based on the mapping relationship between the original data and the coded value in the target fragment dictionary.

[0145] Step 705: The data node performs a query operation on the queried code value in the target shard to obtain the processing result.

[0146] As one possible implementation, data nodes perform query operations on the coded value to be queried within the target shard, and the operation can be performed according to the specific query request. This application provides the following specific implementations of how data nodes perform query operations on the coded value to be queried under different query requests, but these do not constitute a specific limitation.

[0147] In the first optional implementation, the query request may include a filter query request, which includes a dictionary-encoded field. The data node determines the file containing the query code value based on the code value and each file in the target shard. The data node then performs a filter query operation on the query code value in the file to be queried, obtaining the processing result.

[0148] In the second optional implementation, the query request may include an aggregation query request, which contains a dictionary-encoded field. The data node retrieves the aggregation function from the aggregation query request and performs aggregation processing on the target shard according to the aggregation function and the queried encoded value to obtain the processing result.

[0149] In the third optional implementation, the query request may include a grouped query request, which contains a dictionary-encoded field. Data nodes in the target shard group the data according to the queried encoded value to obtain the processing result.

[0150] As one possible implementation, data nodes can also broadcast query requests to each shard, determine whether the shard dictionary contains the queried code value in each shard, and in the target shard containing the queried code value, perform a query operation on the queried code value corresponding to the original data to be queried, and obtain the processing result of the target shard.

[0151] Step 706: The data node decodes the processing result based on the target shard dictionary to obtain the query result.

[0152] The data nodes decode the processing results based on the mapping relationship between the encoded values ​​in the target fragment and the original data to obtain the query results.

[0153] Based on the above Figure 7 In steps 701 to 706, the data node divides the files in a partition into multiple shards based on the time range corresponding to each shard and the storage time of the file. The shard dictionary of each shard in the partition constitutes the complete partition dictionary of the partition. Therefore, when cross-file queries are required, the data node only needs to determine the target shard corresponding to the target shard dictionary containing the original data to be queried, and perform the query operation on the queried encoded value in the target shard to obtain the processing result. Since each shard includes part of the encoded value of the target partition, the target shard can be further refined after the target partition is determined, realizing shard-level filtering during queries. Querying only in the target shard reduces the amount of data to be queried, further improving query efficiency. The data node decodes the processing result based on the target shard dictionary to obtain the query result. Since each shard dictionary is part of the target partition dictionary, the amount of data when decoding based on the target shard dictionary in the complete partition dictionary is further reduced, thereby further improving query efficiency.

[0154] The above describes the complete process of data querying when partitioning files in a database and performing two-level partitioning. The following example describes the data querying process when performing two-level partitioning.

[0155] Figure 10 This is a flowchart illustrating a data query method provided in an embodiment of this application.

[0156] like Figure 10 As shown, the method may include steps 1001 to 1016.

[0157] Step 1001: Data nodes obtain query requests.

[0158] Step 1002: The data node determines whether the query request contains a dictionary-encoded field. If yes, proceed to step 1003, step 1006, or step 1010. If no, proceed to step 1013.

[0159] Step 1003: If the aggregation query request contains a dictionary-encoded field, the data node reads the original data to be queried from the aggregation query request.

[0160] Step 1004: Data nodes in the same shard undergo aggregation query processing according to the aggregation function.

[0161] In each shard, the data node determines the query code value based on the original data to be queried according to the shard dictionary. In the target shard containing the original data to be queried, the query code value is aggregated according to the aggregation function to obtain the aggregation result of the target shard.

[0162] Step 1005: Data nodes decode the aggregation processing results on different shards to obtain the decoding results.

[0163] Step 1005 and above Figure 7 Step 706 is similar to that in the previous embodiment, and will not be described in detail here.

[0164] Step 1006: If the filter query request contains a dictionary-encoded field, the data node reads the original data to be queried from the filter query request.

[0165] Step 1007: Perform shard-level filtering on data nodes.

[0166] The data node determines the target shard corresponding to the target shard dictionary containing the original data to be queried. Based on the shard dictionaries of multiple shards within the target partition, it determines the target shard, further implementing shard-level filtering on top of partition-level filtering. The data node then determines the query code value corresponding to the original data to be queried based on the target shard dictionary.

[0167] Step 1008: Perform file-level filtering on data nodes.

[0168] Data nodes determine the files containing the query code values ​​based on each file in the target shard, achieving file-level filtering during the query. Within the query files, the data nodes perform filtering operations on the query code values ​​to obtain the filtered results.

[0169] Step 1009: The data nodes decode the filtering results in different shards to obtain the decoding results.

[0170] Step 1009 and above Figure 7 Step 706 is similar to that in the previous embodiment, and will not be described in detail here.

[0171] Step 1010: If the group query request contains a dictionary-encoded field, the data node reads the original data to be queried from the group query request.

[0172] Step 1011: Data nodes are grouped in the same shard dictionary according to the code value to be queried.

[0173] In each shard, the data node determines the query code value based on the original data to be queried according to the shard dictionary. In the target shard containing the original data to be queried, the query code value is hashed and grouped to obtain the grouping result in the target shard containing the query code value.

[0174] Step 1012: Data nodes decode the grouping processing results in different shards to obtain the decoding results.

[0175] Step 1012 and above Figure 7 Step 706 is similar to that in the previous embodiment, and will not be described in detail here.

[0176] Step 1013: Perform an index scan on the data nodes.

[0177] Index scans are a common query optimization technique in databases, used to accelerate data retrieval through indexes. An index is a database object used to store the values ​​of one or more columns in a table, allowing for quick location of the corresponding data rows. When a query needs to retrieve specific data from a table, the database system utilizes indexes to improve query speed.

[0178] Step 1014: Data nodes perform data scanning.

[0179] Data scanning includes full scan, incremental scan, and timed scan.

[0180] Step 1015: Data nodes group data.

[0181] Data grouping is a common operation in database queries. The purpose of data grouping is to group records with the same value together. Data grouping is often used in scenarios such as generating summary reports and statistical data.

[0182] Step 1016: Data nodes perform data calculations.

[0183] Data computation refers to the aggregation calculation of data within a group after grouping, such as summation, counting, and averaging.

[0184] Based on the above Figure 10 In steps 1001 to 1016, when data nodes perform aggregation queries, grouping queries, and filtering queries, they process the data directly in the target shard based on the queried encoded value. Compared with the processing method of querying based on the decoded strings of each file in the database, the queried encoded value has a smaller data volume than the string, making the query more efficient.

[0185] The above describes the complete data query process. In order to realize data query based on partition dictionary or shard dictionary, the data node must first build partition dictionary or shard dictionary. The dictionary construction process is described below.

[0186] Figure 11 This is a flowchart illustrating a partition dictionary construction process provided in an embodiment of this application.

[0187] like Figure 11 As shown, the dictionary construction method may include steps 1101 to 1106.

[0188] Step 1101: The data node obtains the dictionary-encoded fields and multiple raw data groups.

[0189] Data nodes obtain dictionary-encoded fields and multiple raw data groups specified by the user when creating the table through interaction using a data definition language (DDL). For example, the raw data group can be a string group.

[0190] Among them, multiple string groups are obtained by evenly dividing the unique values ​​in multiple files in the database, the dictionary encoding field is the field used to construct the dictionary, and the unique value is used to indicate multiple strings that appear at least once in multiple files.

[0191] The fields in the user-created table are as follows:

[0192] CREATE MEASUREMENT mst(

[0193] location SDICT(string)

[0194] direction string

[0195] time int64 )

[0197] WITH ENGINETYPE = columnstore

[0198] SHARDKEY location TYPE hash PRIMARYKEY location,directionl

[0199] The dictionary-encoded field is SDICT, which is mentioned above. The dictionary-encoded field can be used as a subset of the sharding key to achieve balanced management of the partitioned dictionary. For example, the dictionary-encoded field can be used as a subset of the left prefix of the sharding key.

[0200] Step 1102: Based on the original data in multiple files, the data nodes divide the multiple files into multiple partitions corresponding to multiple original data groups.

[0201] Among them, multiple partitions include a first partition, the first partition includes at least one first file, and at least one first file includes a string in the first original data group corresponding to the first partition.

[0202] Data nodes are driven by data writing. Based on the original data in multiple files, they use hash partitioning to divide multiple files into multiple partitions corresponding to multiple pre-created original data groups.

[0203] As one possible implementation, after writing data into the partition, the data node initializes the various modules in the partition.

[0204] Each partition contains multiple modules that work together to implement processes such as data caching, sorting, disk flushing, and index creation, while also supporting timely data retrieval.

[0205] As one possible implementation, the partitioning includes a dictionary encoding manager module, where data nodes initialize key-value mapping tables based on this module. The dictionary encoding manager module is used for the construction and management of dictionary encodings, including building the mapping relationships between dictionary keys and values.

[0206] The dictionary encoding management module is a functional module within a partition, which may also include modules for implementing other functions. Figure 12 A schematic diagram of a partition module provided in an embodiment of this application, such as... Figure 12 As shown, a partition contains the active table module, the snapshot table module, the immutable table module, the duration info module, the index builder module, the file manager module, and the dictionary encoding management module.

[0207] The active table module is used to cache written data. When the data reaches a certain capacity or exceeds a certain time interval, the cached data is passed to the snapshot table module.

[0208] The snapshot table module is used to reorganize the data layout and perform sorting and deduplication operations on cached data.

[0209] The immutable table module is used to manage data file objects, including ordered and unordered data files.

[0210] The retention strategy module is used for data lifecycle management, and implements data aging and deletion based on the time-to-live (TTL) value set in the database table.

[0211] The index generator module is used for index building, including index cache management and index file generation and merging.

[0212] The file manager module is used for background task management of files, including merging files in random order and merging small files into large files.

[0213] Step 1103: The data node creates multiple partition dictionaries for multiple partitions based on the dictionary encoding field.

[0214] The partition dictionary of the first partition is used to indicate the mapping relationship between the encoded values ​​and the original data in the first partition.

[0215] The dictionary encoding management module in the partition of a data node establishes a two-way mapping table between keys and values ​​through the dictionary encoding field. Here, the key is the original data in the partition, and the value is the encoded value mapped from the original data. The two-way mapping table between keys and values ​​is the partition dictionary.

[0216] As one possible implementation, the data node replaces the original data of at least one first file in the first partition with the corresponding encoded value according to the partition dictionary of the first partition, and generates at least one encoded first file.

[0217] As one possible implementation, the snapshot table module, immutable table module, retention strategy module, index generator module, and file manager module in the first partition of the data node manage at least one encoded first file in the first partition.

[0218] Step 1104: Data nodes obtain the file to be written.

[0219] The data node obtains the file to be written and caches the file to be written through the partition's active table module.

[0220] Step 1105: The data node determines the target original data that is not included in the partition dictionary of multiple partitions based on the original data in the file to be written.

[0221] The raw data to be written to the file may include raw data already included in the partition dictionaries of multiple partitions, as well as raw data not included therein. The data node, through the dictionary encoding management module in each partition, determines the target raw data not included in the partition dictionaries of multiple partitions based on the raw data to be written to the file and the raw data included in the partition dictionaries of multiple partitions.

[0222] Step 1106: Based on the dictionary encoding field, the data node adds the target original data and the target encoding value corresponding to the target original data to the partition dictionary of multiple partitions, thereby obtaining the updated partition dictionary of multiple partitions.

[0223] Based on the dictionary encoding field, the data node divides the target original data into partition dictionaries of multiple partitions, adds the target original data to the partition dictionaries of multiple partitions, and adds the target encoding value corresponding to the target original data in a sequentially incrementing manner, thereby obtaining the updated partition dictionaries of multiple partitions.

[0224] Based on the above Figure 11 In steps 1101 to 1106, the data node evenly divides the unique values ​​from multiple files into multiple raw data groups, with each raw data group containing a similar amount of raw data. The multiple files are then divided into multiple partitions corresponding to these raw data groups. Based on the dictionary encoding field, multiple partition dictionaries are established for each partition. In this way, the partition dictionaries of each partition form a complete dictionary. The query efficiency using partition dictionaries is consistent with that using a global dictionary. Different files within the same partition maintain their partition dictionaries without cross-node communication, thus avoiding the high-cost data synchronization network overhead problem associated with global dictionary queries. Furthermore, the similar amount of raw data in each raw data group corresponding to each partition prevents large differences in the data volume of partition dictionaries between different partitions from affecting query efficiency.

[0225] The preceding text described the complete process of constructing a partition dictionary when partitioning files in a database. This application's embodiments do not limit the granularity of file partitioning. For example, files within a partition can be further divided into fragments; a partition may contain multiple fragments, and each fragment contains a portion of the files within the partition. The following will combine... Figure 13 The process of constructing a shard dictionary when performing two-level partitioning of files in a database is described.

[0226] Figure 13 This is a flowchart illustrating a segmented dictionary construction process provided in an embodiment of this application.

[0227] like Figure 13 As shown, the fragmented dictionary construction process may include steps 1301 to 1306.

[0228] Step 1301: The data node obtains the dictionary-encoded fields and multiple raw data groups.

[0229] Step 1301 and above Figure 11 The steps in step 1101 are similar, and will not be described in detail in this embodiment.

[0230] Step 1302: Based on the original data in multiple files, the data node divides the multiple files into multiple partitions corresponding to multiple original data groups.

[0231] Step 1302 and above Figure 11 Step 1102 is similar to that in the previous embodiment, and will not be described in detail in this application.

[0232] Step 1303: Based on the storage time of at least one first file, the data node divides at least one first file into multiple shards in the first partition.

[0233] Each of the multiple shards is used to store the first file whose file storage time belongs to the corresponding time range.

[0234] Step 1304: The data nodes create a shard dictionary for each shard based on the dictionary encoding field.

[0235] The first segment's segment dictionary is used to indicate the mapping relationship between the encoded values ​​in the first segment and the original data.

[0236] As one possible implementation, the dictionary encoding management module in the first partition of the data node establishes a bidirectional mapping table between the key-value pairs of the first partition through the dictionary encoding field.

[0237] Step 1305: The data node replaces the string of the first file in the first segment with the corresponding encoded value according to the segment dictionary of the first segment, and generates the encoded first file.

[0238] The data node maps the original data of the first file in the first segment to the encoded value through the segment dictionary of the first segment, and replaces the original data with the encoded value as the encoded first file generated by data encoding.

[0239] As one possible implementation, the snapshot table module, immutable table module, retention strategy module, index generator module, and file manager module in the first partition of the data node manage the encoded first file in the first partition.

[0240] Step 1306: If the storage duration of the file with the latest storage time contained in the first shard exceeds the preset duration threshold, the data node deletes the first shard.

[0241] The data node determines the storage duration of the file with the latest storage time based on the storage time of the files contained in the first shard. If the storage duration exceeds the preset duration threshold, the first shard is deleted.

[0242] As one possible implementation, the retention policy module in the first partition of the data node deletes the files and the partition dictionary in the first partition based on the storage time of the files contained in the first partition and the preset time-to-live value. This achieves data aging deletion.

[0243] Based on the above Figure 13 Steps 1301 to 1306 describe how data nodes divide files within the same partition into different shards based on their storage time, thereby achieving fine-grained management of files based on their time attributes and improving query efficiency when performing queries based on shards. A shard dictionary is built for each shard based on the dictionary encoding field. The shard dictionary for the first shard indicates the mapping relationship between strings and encoded values ​​within the first shard. Therefore, a shard only contains a portion of the files in the partition. When querying based on the shard dictionary, the amount of data retrieved precisely down to the shard level is relatively small, thus improving query efficiency.

[0244] The data query method provided in this embodiment has been described in detail above. The following will combine... Figure 14 This describes the data query device provided in this embodiment.

[0245] Figure 14 This is a schematic diagram of a data query device 1400 provided in an embodiment of this application. The data query device 1400 can be used to implement the function of the data node in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the data query device 1400 can be a data node, or it can be a module (such as a chip) applied to the data node.

[0246] The data query device 1400 includes an acquisition module 1401, a query module 1402, and a decoding module 1403.

[0247] The acquisition module 1401 is used to acquire a query request, which contains the original data to be queried and the query operation. For example, the acquisition module 1401 is used to execute... Figure 5 Step 501 or execution Figure 7 Step 701 in the process.

[0248] The query module 1402 is used to determine the target partition containing the original data to be queried, based on the original data to be queried and the partition dictionaries of multiple partitions. Each partition dictionary contains a mapping relationship between the encoded values ​​in each partition and the original data. Based on the target partition dictionary of the target partition, the query encoded value corresponding to the original data to be queried is determined. In the target partition, a query operation is performed on the query encoded value to obtain the processing result. For example, the query module 1402 is used to execute... Figure 5Steps 502 to 504 or execution Figure 7 Steps 702 to 705 in the process.

[0249] Decoding module 1403 is used to decode the processing results based on the target partition dictionary to obtain the query results. For example, decoding module 1403 is used to execute... Figure 5 Step 505 or execution Figure 7 Step 706 in the process.

[0250] As one possible implementation, the query request includes a filtered query request, which contains a dictionary-encoded field. Specifically, the query module 1402 is used to determine the file containing the query code value based on the query code value and each file in the target partition; within the file to be queried, a filtered query operation is performed on the query code value to obtain the processing result.

[0251] As one possible implementation, the query request includes an aggregate query request, which contains a dictionary-encoded field. The query module 1402 is specifically used to obtain the aggregate function from the aggregate query request; and to perform aggregation processing on the target partition according to the aggregate function and the coded value to be queried, to obtain the processing result.

[0252] As one possible implementation, the query request includes a grouped query request, which contains a dictionary-encoded field. Specifically, the query module 1402 is used to group the target partition according to the queried encoded value to obtain the processing result.

[0253] As one possible implementation, the target partition includes multiple shards, each of which includes a portion of the target partition's encoded value. Each shard is partitioned based on the file storage time of the files in the target partition. The query module 1402 is further configured to determine the target shard containing the original data to be queried, based on the original data to be queried and the shard dictionary corresponding to each shard. Each shard dictionary indicates a partial mapping relationship of the target partition dictionary, and each shard dictionary contains a mapping relationship between a portion of the encoded value of each shard and the original data. Specifically, it is configured to determine the encoded value to be queried corresponding to the original data to be queried based on the target shard dictionary of the target shard; perform a query operation on the encoded value to be queried in the target shard to obtain a processing result; and decode the processing result based on the target shard dictionary to obtain the query result.

[0254] As one possible implementation, the query module 1402 is further configured to delete the target fragment dictionary from the target partition dictionary when the storage duration of the target fragment in the target partition exceeds a preset duration threshold.

[0255] As one possible implementation, the acquisition module 1401 is also used to acquire the file to be written;

[0256] The query module 1402 is further configured to determine, based on the original data in the file to be written, the target original data not included in the partition dictionary of the multiple partitions;

[0257] The device further includes an update module 1404, which is used to add the target original data and the target encoding value corresponding to the target original data to the partition dictionary of the multiple partitions based on the dictionary encoding field, so as to obtain an updated partition dictionary of the multiple partitions.

[0258] The acquisition module 1401, query module 1402, and decoding module 1403 can all be implemented in software or in hardware. For example, the implementation of the acquisition module 1401 will be described below. Similarly, the implementation of the query module 1402 and decoding module 1403 can refer to the implementation of the acquisition module 1401.

[0259] It is understood that the specific implementation methods of each module of the data query device 1400 for executing the data query method can refer to the specific steps in the above method embodiments, and will not be repeated here.

[0260] As an example of a software functional unit, module 1401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 1401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0261] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0262] As an example of a hardware functional unit, the acquisition module 1401 may include at least one computing device, such as a server. Alternatively, the acquisition module 1401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0263] The multiple computing devices included in the acquisition module 1401 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 1401 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 1401 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0264] It should be noted that, in other embodiments, the acquisition module 1401 can be used to execute any step in the data query method, the query module 1402 can be used to execute any step in the data query method, and the decoding module 1403 can be used to execute any step in the data query method. The steps implemented by the acquisition module 1401, the query module 1402, and the decoding module 1403 can be specified as needed. By implementing different steps in the data query method through the acquisition module 1401, the query module 1402, and the decoding module 1403, all functions of the data query device 1400 can be realized.

[0265] This application also provides a computing device 1500. Figure 15This is a schematic diagram of the structure of a computing device provided in an embodiment of this application, such as... Figure 15 As shown, the computing device 1500 includes a bus 1502, a processor 1504, a memory 1506, and a communication interface 1508. The processor 1504, the memory 1506, and the communication interface 1508 communicate with each other via the bus 1502. The computing device 1500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0266] Bus 1502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 15 The bus 1502 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1502 may include a path for transmitting information between various components of the computing device 1500 (e.g., memory 1506, processor 1504, communication interface 1508).

[0267] Processor 1504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0268] The memory 1506 may include volatile memory, such as random access memory (RAM). The processor 1504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0269] The memory 1506 stores executable program code, which the processor 1504 executes to implement the functions of the aforementioned acquisition module 1401, query module 1402, and decoding module 1403, thereby realizing the data query method. In other words, the memory 1506 stores instructions for executing the data query method.

[0270] The communication interface 1508 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1500 and other devices or communication networks.

[0271] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0272] Figure 16 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application, such as... Figure 16 As shown, the computing device cluster includes at least one computing device 1500. The memory 1506 of one or more computing devices 1500 in the computing device cluster may store the same instructions for executing data query methods.

[0273] In some possible implementations, the memory 1506 of one or more computing devices 1500 in the computing device cluster may also store partial instructions for executing the data query method. In other words, a combination of one or more computing devices 1500 can jointly execute the instructions for executing the data query method.

[0274] It should be noted that the memory 1506 in different computing devices 1500 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the data query device 1400. That is, the instructions stored in the memory 1506 of different computing devices 1500 can implement the functions of one or more modules among the acquisition module 1401, query module 1402, and decoding module 1403.

[0275] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 17 This is a schematic diagram illustrating a network connection structure between computing devices, as provided in an embodiment of this application. Figure 17 As shown, the two computing devices 1500A and 1500B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1506 in computing device 1500A stores instructions for executing the functions of the acquisition module 1401. Simultaneously, the memory 1506 in computing device 1500B stores instructions for executing the functions of the query module 1402 and the decoding module 1403.

[0276] Figure 17 The connection method between the computing device clusters shown can be that, considering that the data query method provided in this application needs to store query requests, the function implemented by the acquisition module 1401 is delegated to the computing device 1500A for execution.

[0277] It should be understood that Figure 17 The functions of computing device 1500A shown can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0278] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a data query method.

[0279] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a data query method.

[0280] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data query method, characterized in that, The method is used to query a database, which includes multiple partitions. Each partition includes encoded values ​​corresponding to a portion of the original data in the database. The method includes: Obtain a query request, which includes the original data to be queried and the query operation; Based on the original data to be queried and the partition dictionaries of the multiple partitions, the target partition containing the original data to be queried is determined, wherein each partition dictionary contains the mapping relationship between the encoded values ​​in each partition and the original data; Based on the target partition dictionary of the target partition, determine the query code value corresponding to the original data to be queried; In the target partition, the query operation is performed on the code value to be queried to obtain the processing result; The processing result is decoded based on the target partition dictionary to obtain the query result.

2. The method according to claim 1, characterized in that, The query request includes a filtering query request, which contains a dictionary-encoded field. The query operation is performed on the encoded value to be queried in the target partition to obtain a processing result, including: Based on the query code value and each file in the target partition, determine the query file containing the query code value; In the file to be queried, the queried code value is filtered and queried to obtain the processing result.

3. The method according to claim 1, characterized in that, The query request includes an aggregate query request, which contains a dictionary-encoded field. The query operation is performed on the queried encoded value in the target partition to obtain the processing result, including: Retrieve the aggregate function from the aggregate query request; The target partition is aggregated according to the aggregation function and the queried code value to obtain the processing result.

4. The method according to claim 1, characterized in that, The query request includes a grouped query request, which contains a dictionary-encoded field. The query operation is performed on the queried encoded value in the target partition to obtain the processing result, including: The target partition is grouped according to the query code value to obtain the processing result.

5. The method according to any one of claims 1-4, characterized in that, The target partition comprises multiple fragments, each fragment including a portion of the target partition's encoded value, and each fragment is obtained based on the file storage time of the files in the target partition; after determining the target partition containing the original data to be queried based on the original data to be queried and the partition dictionaries of the multiple partitions, the method further includes: Based on the original data to be queried and the shard dictionary corresponding to each shard, a target shard containing the original data to be queried is determined, wherein each shard dictionary is used to indicate a partial mapping relationship of the target partition dictionary, and each shard dictionary contains a mapping relationship between a partial encoded value of each shard and the original data; The step of determining the query code value corresponding to the original data to be queried based on the target partition dictionary of the target partition includes: Based on the target fragment dictionary of the target fragment, determine the query code value corresponding to the original data to be queried; The process of performing the query operation on the target partition for the code value to be queried, and obtaining the processing result, includes: In the target segment, the query operation is performed on the queried encoded value to obtain the processing result; The process of decoding the processing result based on the target partition dictionary yields the query result, including... The processing result is decoded based on the target fragment dictionary to obtain the query result.

6. The method according to claim 5, characterized in that, When the storage duration of the target fragment in the target partition exceeds a preset duration threshold, the target fragment dictionary is deleted from the target partition dictionary.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Get the file to be written; Based on the original data in the file to be written, determine the target original data that is not included in the partition dictionary of the multiple partitions; Based on the dictionary encoding field, the target original data and the target encoding value corresponding to the target original data are added to the partition dictionary of the multiple partitions to obtain the updated partition dictionary of the multiple partitions.

8. A data query device, characterized in that, The device includes: The acquisition module is used to acquire query requests, which include the original data to be queried and the query operation. The query module is used to determine the target partition containing the original data to be queried, based on the original data to be queried and the partition dictionaries of multiple partitions. The multiple partitions are multiple partitions of a database, each partition includes an encoded value corresponding to a portion of the original data in the database, and each partition dictionary contains a mapping relationship between the encoded values ​​in each partition and the original data. Based on the target partition dictionary of the target partition, the module determines the queried encoded value corresponding to the original data to be queried. The module then performs the query operation on the queried encoded value in the target partition to obtain a processing result. The decoding module is used to decode the processing result based on the target partition dictionary to obtain the query result.

9. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

10. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-7.

Images