Adaptive hybrid sorting method and system based on hash bitmap and quick sort

Abstract

The present application belongs to the field of databases and specifically discloses an adaptive hybrid sorting method and system based on hash bitmap and quick sort. The method includes: obtaining row identifiers that meet the query conditions from a database index to obtain a query result set; splitting the row identifier into a compression unit identifier and a row number, storing the compression unit identifier in an identifier array, and storing the row number in a hash bitmap; using a hybrid sorting method to quickly sort the identifier array to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on hash bitmap and quick sort; traversing the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; splicing the compression unit identifier and the compression unit bitmap to obtain a complete row identifier, so as to return a result set based on the complete row identifier for recording. The present application can improve query performance and efficiency.

Description

Technical Field

[0001] The present application relates to the field of databases, and more specifically, to an adaptive hybrid sorting method and system based on hash bitmap and quick sorting. Background Art

[0002] In database systems, indexes are an important means of improving query performance. When a query returns a large result set with many rows but not large enough to process all table data, the optimizer typically balances between index scans and sequential scans. Because index scans introduce a large number of random I / O operations, which are more expensive than sequential scans, the optimizer typically chooses a bitmap scan to reduce the cost of the execution plan.

[0003] However, traditional column-stored table bitmap scans use ordered CTID vector arrays instead of bitmaps. While this solves the memory waste problem when the result set is sparse, it also introduces technical issues such as high sorting time complexity, high memory usage, and large external sorting overhead. These issues are particularly prominent in large-scale OLAP query scenarios, resulting in poor database query performance and efficiency, making further improvement difficult.

[0004] Therefore, an efficient column-stored table bitmap scanning optimization solution is urgently needed to improve the query performance and efficiency of the database. Summary of the Invention

[0005] In view of the defects of the prior art, the purpose of this application is to provide an adaptive hybrid sorting method and system based on hash bitmap and quick sort, aiming to solve the problem of poor query performance of database systems in the prior art.

[0006] To achieve the above objectives, in a first aspect, the present application provides an adaptive hybrid sorting method based on hash bitmap and quick sort, which is applied to a column-stored table of a database, comprising:

[0007] Obtain row identifiers that meet the query conditions from the database index to obtain the query result set;

[0008] Splitting the row identifier into a compression unit identifier and a row number, storing the compression unit identifier in an identifier array, and storing the row number in a hash bitmap;

[0009] Using a hybrid sorting method to quickly sort the identifier array to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting;

[0010] Traversing the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap;

[0011] A complete row identifier is obtained by concatenating the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

[0012] Optionally, the concatenation method of the complete row identifier includes:

[0013] Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap;

[0014] A complete row identifier is obtained by concatenating the compression unit identifier and the valid row number, so as to read and return data records in a sequence based on the complete row identifier.

[0015] Optionally, it also includes:

[0016] Select a sorting method based on the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy;

[0017] In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold;

[0018] In the case of a memory-first strategy, obtain the ratio between the amount of data in the result set and the total number of records in the table, and select a sorting method based on the ratio and the pre-qualified ratio threshold.

[0019] Optionally, the identifier array stores only the deduplicated compression unit identifiers in the result set;

[0020] Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap;

[0021] Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the record is filtered.

[0022] Optionally, the method of storing the row number in the hash bitmap includes:

[0023] Using the compression unit identifier as a key, the row number corresponding to the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

[0024] Optionally, the method for determining the sorting method includes:

[0025] If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed the preset threshold, a row identifier vector array sorting method is selected;

[0026] If the proportion exceeds the proportion threshold, a hash bitmap and quick sort hybrid sorting method is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

[0027] Optionally, the row identifier vector array sorting method includes:

[0028] Store the row identifiers obtained from the query into the row identifier array;

[0029] Sort the row identifier array to obtain an array sequence;

[0030] Traverse the sorted array sequence, access the compression unit corresponding to each row identifier in sequence, and read the corresponding data record from the compression unit according to the row number.

[0031] In a second aspect, the present application further provides an adaptive hybrid sorting system based on hash bitmap and quick sort, comprising:

[0032] The query module is used to obtain row identifiers that meet the query conditions from the database index and obtain the query result set;

[0033] a splitting module, configured to split the row identifier into a compression unit identifier and a row number, store the compression unit identifier in an identifier array, and store the row number in a hash bitmap;

[0034] a sorting module, configured to perform quick sorting on the identifier array using a hybrid sorting method to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting;

[0035] A traversal module, configured to traverse the array sequence and determine a compression unit bitmap corresponding to each compression unit identifier in the hash bitmap;

[0036] The splicing module is used to obtain a complete row identifier according to the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

[0037] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0039] In a fifth aspect, the present application provides a computer program product, which, when executed on a processor, enables the processor to execute the method described in the first aspect or any possible implementation of the first aspect.

[0040] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0041] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the existing technologies:

[0042] (1) This application splits the row identifier into a compression unit identifier and a row number, and stores them in an array and a hash bitmap respectively, effectively reducing the size of the data to be sorted. The compression unit identifier is used as a frequently accessed primary key and is stored independently for easy sorting, while the row number is stored in a compact binary form through a hash bitmap, saving continuous memory space. The split storage method not only reduces memory usage but also avoids the duplication of redundant data in full sorting, thereby significantly reducing sorting time and improving the query performance and efficiency of the database.

[0043] (2) This application further optimizes the sorting performance by introducing a hybrid sorting method that combines the advantages of hash bitmap and quick sort. The hash bitmap pre-groups the row numbers associated with the compression unit identifiers, avoiding the frequent cross-compression unit identifier comparison operations in traditional full sorting, and significantly reducing the sorting complexity. Quick sort performs local sorting on the compression unit identifiers after bucketing, taking into account both global orderliness and local efficiency. The hybrid sorting method not only adapts to sparse or dense data distribution characteristics, but also maintains stable performance under extreme data distribution, thereby further improving sorting efficiency.

[0044] (3) This application can select the appropriate sorting method based on performance and memory requirements, allowing the system to flexibly adjust resource usage and optimize performance when processing data of different sizes. The identifier array only stores deduplicated compression unit identifiers, effectively reducing memory usage and improving data processing efficiency.

[0045] (4) This application dynamically selects a sorting method based on the amount of data in the query result set and the proportion of the total number of table records, thereby improving the algorithm's adaptability and enabling it to maintain good performance in different usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is one of the flow charts of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application;

[0047] Figure 2This is the second flow chart of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application;

[0048] Figure 3 This is the third flow chart of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application;

[0049] Figure 4 This is a flowchart of the traditional CTID array qsort sorting process;

[0050] Figure 5 This is the fourth flow chart of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application;

[0051] Figure 6 Schematic diagram of the structure of an adaptive hybrid sorting system based on hash bitmap and quick sort provided in an embodiment of the present application;

[0052] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0055] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0056] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0058] First, the terms of the embodiments of the present application are explained in detail.

[0059] Index: A structure that sorts the values ​​of one or more columns in a database table. Using an index can quickly access specific information in a database table and is a means of optimizing database performance.

[0060] Row-based storage, also known as row-based storage or horizontal storage, is a traditional database storage method. In this method, data is organized into rows, which are logical storage units. Each row contains several fields or columns, which are arranged in a specific order.

[0061] Column-based storage, also known as column storage or vertical storage, is an emerging database storage method. In this method, data is organized using columns as the underlying logical storage unit. Each column contains several fields or values, which are compressed and stored in units of CUs. A column can have multiple CUs.

[0062] CU (Compression Unit): A compression unit is the smallest storage unit for column-level data. Some column-level databases also call it a segment. Typically, a CU is composed of tens of thousands of compressed records in a single field.

[0063] cuid: CU number, that is, compression unit identifier.

[0064] CTID: Row identifier, a system column provided by PostgreSQL. Its field type is TID (Tuple Identifier). It uniquely identifies each record and identifies the physical storage location of each row in the table. The CTID for row-oriented tables consists of two parts: the block number (4 bytes) and the row number (2 bytes). For example, (0, 10) represents the 10th row in physical block 0. The CTID for column-oriented tables consists of the CUID (4 bytes) and the row number (2 bytes).

[0065] qsort (Quick Sort): A C standard library function based on the quick sort algorithm. Its average execution time is O(NlogN), but in certain cases (such as for an already sorted array), the time complexity can degenerate to O(N^2).

[0066] Quick sort: Using the divide-and-conquer idea, an arbitrary pivot element is selected from an unordered sequence, and the sequence to be sorted is divided into two parts using the pivot. The elements in the front part are all less than or equal to the pivot element, and the elements in the back part are all greater than or equal to the pivot element. Then, a recursive method is used to repeat the above operations on the front and back parts respectively until the unordered sequence is arranged into an ordered sequence.

[0067] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0068] Reference Figure 1 The present application provides an adaptive hybrid sorting method based on hash bitmap and quick sort, which is applied to a column-stored table of a database, including:

[0069] S101. Obtain row identifiers that meet the query conditions from the database index and obtain the query result set;

[0070] S102. Split the row identifier into a compression unit identifier and a row number, store the compression unit identifier in an identifier array, and store the row number in a hash bitmap;

[0071] S103. Perform quick sorting on the identifier array using a hybrid sorting method to obtain an array sequence of compression unit identifiers; the hybrid sorting method is based on a hash bitmap and quick sorting implementation;

[0072] S104. Traverse the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap;

[0073] S105. Obtain a complete row identifier based on the compression unit identifier and the compression unit bitmap, and return a result set based on the complete row identifier for recording.

[0074] First, it should be noted that the embodiment of the present application is applied to the optimization of the bitmap scan operator of the column-stored table.

[0075] In step S101, row identifiers that meet the query criteria are retrieved from the database index, resulting in a query result set. The core of this step is to quickly locate rows that meet the query criteria using the index. Indexes, as key data structures for efficient database queries, can significantly reduce the overhead of full table scans. After obtaining row identifiers through the index, these identifiers represent the physical location of the qualifying rows in the database, providing a foundation for subsequent processing.

[0076] It should be noted that the query conditions in the embodiments of the present application are filtering rules set by the user or program when querying data, which are used to find data that meets the requirements. Common query conditions include:

[0077] Equal to a certain value: For example, filter out the value of a column that is equal to specific data.

[0078] Within a certain range: For example, filter out values ​​in a column that are greater than or less than a certain number, or within a certain interval.

[0079] Fuzzy matching: For example, filtering out values ​​in a column that match a certain pattern, such as starting with a certain word.

[0080] Combination conditions: For example, multiple conditions must be met at the same time, or one of the conditions must be met.

[0081] Null value check: For example, filter out the value of a column to be empty or not empty.

[0082] In a certain collection: For example, filter out the values ​​of a column that belong to a specific collection.

[0083] In the embodiments of this application, query conditions are first used to find matching row identifiers from the database index. These row identifiers are then split into two parts: a compression unit identifier and a row number, each stored in a different data structure. Then, through sorting and bitmap processing, a complete result set is generated and returned to the user. In other words, query conditions are the rules used to filter data and form the basis for obtaining a result set.

[0084] In step S102, the row identifier is split into a compression unit identifier and a row number, and the compression unit identifier is stored in an identifier array, and the row number is stored in a hash bitmap.

[0085] A row identifier typically consists of two pieces of information: a compression unit identifier, which identifies the compression unit in which the data resides, and a row number, which identifies the specific row within the compression unit. With split storage, compression unit identifiers are stored centrally in an array, facilitating subsequent sorting operations, while row numbers are stored in a compact binary format using a hash bitmap, saving memory space and enabling fast locating.

[0086] Reference Figure 2 , Figure 2 This is a schematic diagram of the data structure of the hybrid sorting method HBQSAH sorting in the embodiment of the present application.

[0087] In step S103, the identifier array is quickly sorted using a hybrid sorting method to obtain an array sequence of compression unit identifiers.

[0088] The hybrid sorting method combines the advantages of hash bitmaps and quick sort. First, the row numbers associated with compression unit identifiers are pre-grouped using a hash bitmap, avoiding the frequent cross-compression unit identifier comparisons required in traditional full sorting and reducing sorting complexity. Quick sorting is then performed on the bucketed compression unit identifiers, balancing global ordering with local efficiency, ensuring stable sorting performance across diverse data distributions.

[0089] In step S104, the sorted array sequence is traversed to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap. The hash bitmap is used to quickly locate the row number corresponding to each compression unit identifier, avoiding the overhead of parsing the compression unit metadata row by row, further shortening data extraction time.

[0090] In step S105, a complete row identifier is obtained by concatenating the compression unit identifier and the compression unit bitmap, and a result set is returned based on the complete row identifier for recording.

[0091] The splicing process recombines the compression unit identifier with the row number in the hash bitmap to generate a complete row identifier, ensuring the accuracy and completeness of the result set. By splicing and streaming results on demand, query results are returned gradually, avoiding waiting delays before the full sort is completed. This also reduces network and memory bandwidth pressure, significantly improving query response speed.

[0092] Optionally, the concatenation method of the complete row identifier includes:

[0093] Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap;

[0094] A complete row identifier is obtained by concatenating the compression unit identifier and the valid row number, so as to read and return data records in a sequence based on the complete row identifier.

[0095] Specifically, first calculate the hash value of the compression unit identifier and extract the corresponding compression unit bitmap from the hash bitmap. This step quickly locates the target bitmap through hash mapping, providing a basis for subsequent parsing.

[0096] The compression unit bitmap is then parsed, and row numbers marked as valid are traversed bit by bit to identify them, generating a list of valid row numbers. This process significantly reduces storage and computational overhead through the efficient bitmap representation. The compression unit identifier is then concatenated with the valid row number to generate a complete row identifier. This step ensures the precise location of data records through the construction of a unique identifier.

[0097] Finally, based on the order of the complete row identifiers, the corresponding data records are read and returned from the data storage system.

[0098] For example, if cuid=100, the bitmap is 000101, and each bit is checked from back to front. If it is =1, it is a valid bit, and its position is the corresponding row number, which is 1 and 3. The ctid of the spliced ​​output is (100, 1) and (100, 3).

[0099] This process ensures data consistency and continuity through sequential reading, while also improving retrieval efficiency. The entire solution leverages key technologies such as hash bitmaps, bitmap parsing, and identifier concatenation to achieve efficient and flexible data record access, making it suitable for large-scale data storage and retrieval scenarios.

[0100] Reference Figure 3 , Figure 3 This is a complete flowchart of the application embodiment, including the following steps:

[0101] First, search the index for a ctid that meets the conditions and split the found ctid into a cuid array and a hash bitmap. Next, determine whether the cuid array is empty. If it is, obtain the next ctid that meets the conditions and perform qrsort on the cuid array. If it is empty, terminate the process. After the sort is complete, traverse the cuid array, obtain the next cuid, and determine whether it is empty. If it is, load the CU data based on the cuid and traverse the bitmap corresponding to the cuid to obtain the next valid bit of the bitmap. If it is empty, terminate the process. While traversing the bitmap, concatenate the ctid based on the cuid and the bitmap, and return the record.

[0102] Optionally, it also includes:

[0103] Select a sorting method based on the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy;

[0104] In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold;

[0105] In the case of a memory-first strategy, obtain the ratio between the amount of data in the result set and the total number of records in the table, and select a sorting method based on the ratio and the pre-qualified ratio threshold.

[0106] Furthermore, the method for determining the sorting method includes:

[0107] If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed the preset threshold, a row identifier vector array sorting method is selected;

[0108] If the proportion exceeds the proportion threshold, a hash bitmap and quick sort hybrid sorting method is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

[0109] Specifically, the embodiment of the present application can adaptively select whether the operator uses traditional CTID vector array sorting or HBQSAH sorting based on the estimated data volume of the result set and the ratio of the total number of records in the table and the sorting optimization strategy (performance priority, memory priority).

[0110] If the performance priority strategy is used, if the estimated result set data volume exceeds the specified threshold (default 1000), HBQSAH sorting is used; otherwise, CTID vector array sorting is used;

[0111] If the memory-first strategy is used, and the estimated result set data volume accounts for more than 2% of the total number of records in the table, HBQSAH sorting is used; otherwise, CTID vector array sorting is used.

[0112] The following is a specific implementation case. The detailed implementation steps are as follows:

[0113] A column-based table for an OLAP application has 200 fields, 50 million records, and 6.9 GB of data. 833 CUs contain 60,000 records, and one CU contains 20,000 records.

[0114] 1) Set the performance priority policy, with the default threshold of 1000;

[0115] The result set is 1000, so choose traditional qsort.

[0116] The result set is 10,000, so select HBQSAH sorting;

[0117] If the result set is around 100,000, select HBQSAH sort.

[0118] 2) Set the performance priority policy with a threshold of 20,000;

[0119] The result set is 10,000, so choose traditional qsort.

[0120] If the result set is around 100,000, select HBQSAH sort.

[0121] 3) Set a memory priority policy with a threshold of 2% of the total number of records in the table;

[0122] The result set is 10,000, so choose traditional qsort.

[0123] If the result set is 100,000, choose traditional qsort.

[0124] If the result set is over 1 million, select HBQSAH sort.

[0125] Optionally, the identifier array stores only the deduplicated compression unit identifiers in the result set;

[0126] Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap;

[0127] Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the record is filtered.

[0128] Optionally, the method of storing the row number in the hash bitmap includes:

[0129] Using the compression unit identifier as a key, the row number corresponding to the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

[0130] Specifically, the sorting method of the embodiment of the present application replaces the CTID vector array with a cuid array and a bitmap hash table. The cuid array only stores the cuids of the result set without duplication. Each member of the bitmap hash table consists of cuid, bitmap size and CU bit. Figure 3 The CU bitmap is composed of 10 fields. cuid is used as the key in the hash table for fast bitmap queries. The CU bitmap marks the row number of the records that meet the conditions in the CU. Each bit in the bitmap represents a row of records in the CU. A value of 1 indicates that the record meets the conditions, and a value of 0 indicates that the record is filtered out.

[0131] Optionally, the row identifier vector array sorting method includes:

[0132] Store the row identifiers obtained from the query into the row identifier array;

[0133] Sort the row identifier array to obtain an array sequence;

[0134] Traverse the sorted array sequence, access the compression unit corresponding to each row identifier in sequence, and read the corresponding data record from the compression unit according to the row number.

[0135] Specifically, refer to Figure 4 , Figure 4 This is a schematic diagram of the traditional CTID array qsort sorting process, which includes the following steps:

[0136] First, the index searches for a ctid that meets the criteria and places it into the ctid array. Next, the ctid array is checked to see if it is empty. If so, the next ctid that meets the criteria is retrieved and sorted using QSort. If the array is empty, the process ends immediately. After sorting, the ctid array is traversed, the next ctid is retrieved, and the array is checked to see if it is empty. If so, the record is returned based on the ctid. If not, the process ends. This entire process, through step-by-step searching, sorting, and traversal, ultimately returns the record that meets the criteria.

[0137] Further, refer to Figure 5 , Figure 5 : This is a schematic diagram of the Index Ctid Scan operator adaptive hybrid sorting process in an embodiment of the present application, including the following steps:

[0138] First, decide whether to use a sorting algorithm based on the estimated number of result sets and the priority strategy. If sorting is used, an index scan is performed, all CTIDs that meet the criteria are placed in an array, the CTID array is sorted, and data is retrieved from the CU based on the sorted CTID array. If sorting is not used, an index scan is performed, all CTIDs that meet the criteria are split and placed in a hash bitmap, the cuid array is sorted, and data is retrieved from the CU based on the hash bitmap.

[0139] Based on the estimated ratio of the result set's data volume to the total number of records in the table and the sorting optimization strategy (performance first, memory first), the operator is adaptively selected to use traditional CTID vector array sorting or HBQSAH sorting.

[0140] If you choose the traditional CTID vector array sort, the original process is executed and the result set is returned;

[0141] If HBQSAH sorting is selected, first scan the index for all rows that meet the conditions, split the CTID into cuid and row number, and put them into the cuid array and bitmap hash respectively. The cuid array is deduplicated. When this step is completed, the row numbers in the bitmap are in order, and only the cuid array needs to be sorted.

[0142] Sort the cuid array qsort;

[0143] Load CU in the order of cuid array, generate ctid based on the valid bits in bitmap and cuid, and get the corresponding records in CU in sequence.

[0144] Reference Figure 6 , the present application also provides an adaptive hybrid sorting system based on hash bitmap and quick sort, including:

[0145] Query module 610, used to obtain row identifiers that meet the query conditions from the database index to obtain the query result set;

[0146] a splitting module 620, configured to split the row identifier into a compression unit identifier and a row number, store the compression unit identifier in an identifier array, and store the row number in a hash bitmap;

[0147] a sorting module 630 for performing quick sorting on the identifier array using a hybrid sorting method to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting;

[0148] A traversal module 640 is configured to traverse the array sequence and determine a compression unit bitmap corresponding to each compression unit identifier in the hash bitmap;

[0149] The splicing module 650 is configured to obtain a complete row identifier by splicing the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

[0150] Optionally, the concatenation method of the complete row identifier includes:

[0151] Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap;

[0152] A complete row identifier is obtained by concatenating the compression unit identifier and the valid row number, so as to read and return data records in a sequence based on the complete row identifier.

[0153] Optionally, a sorting method selection module is further included, for:

[0154] Select a sorting method based on the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy;

[0155] In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold;

[0156] In the case of a memory-first strategy, obtain the ratio between the amount of data in the result set and the total number of records in the table, and select a sorting method based on the ratio and the pre-qualified ratio threshold.

[0157] Optionally, the identifier array stores only the deduplicated compression unit identifiers in the result set;

[0158] Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap;

[0159] Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the record is filtered.

[0160] Optionally, the method of storing the row number in the hash bitmap includes:

[0161] Using the compression unit identifier as a key, the row number corresponding to the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

[0162] Optionally, the method for determining the sorting method includes:

[0163] If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed the preset threshold, a row identifier vector array sorting method is selected;

[0164] If the proportion exceeds the proportion threshold, a hash bitmap and quick sort hybrid sorting method is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

[0165] Optionally, the row identifier vector array sorting method includes:

[0166] Store the row identifiers obtained from the query into the row identifier array;

[0167] Sort the row identifier array to obtain an array sequence;

[0168] Traverse the sorted array sequence, access the compression unit corresponding to each row identifier in sequence, and read the corresponding data record from the compression unit according to the row number.

[0169] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0170] Reference Figure 7 Based on the methods in the above embodiments, an embodiment of the present application provides an electronic device, which may include: a processor (Processor) 710, a communication interface (Communications Interface) 720, a memory (Memory) 730, and a communication bus 740. The processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call logic instructions in the memory 730 to execute the methods in the above embodiments.

[0171] In addition, the logic instructions in the aforementioned memory 730 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0172] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0173] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0174] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0175] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC.

[0176] The above embodiments can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions. When loaded and executed on a computer, the computer program instructions fully or partially produce the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)).

[0177] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0178] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An adaptive hybrid sorting method based on hash bitmap and quick sort, characterized in that: Column-stored tables used in databases include: Obtain row identifiers that meet the query conditions from the database index to obtain the query result set; Splitting the row identifier into a compression unit identifier and a row number, storing the compression unit identifier in an identifier array, and storing the row number in a hash bitmap; Using a hybrid sorting method to quickly sort the identifier array to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; Traversing the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; obtaining a complete row identifier by concatenating the compression unit identifier and the compression unit bitmap, and returning a result set based on the complete row identifier for recording; Select a sorting method based on the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy; In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold; In the case of a memory-first strategy, obtain the ratio between the amount of data in the result set and the total number of records in the table, and select a sorting method based on the ratio and the pre-qualified ratio threshold.

2. The adaptive hybrid sorting method based on hash bitmap and quick sort according to claim 1, characterized in that: The concatenation methods for complete row identifiers include: Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap; A complete row identifier is obtained by concatenating the compression unit identifier and the valid row number, so as to read and return data records in a sequence based on the complete row identifier.

3. The adaptive hybrid sorting method based on hash bitmap and quick sort according to claim 1, characterized in that: The identifier array only stores the deduplicated compression unit identifiers in the result set; Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap; Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the record is filtered.

4. The adaptive hybrid sorting method based on hash bitmap and quick sort according to claim 1, characterized in that: Methods for storing row numbers in a hash bitmap include: Using the compression unit identifier as a key, the row number corresponding to the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

5. The adaptive hybrid sorting method based on hash bitmap and quick sort according to claim 1, characterized in that: The method for determining the sorting method includes: If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed the preset threshold, a row identifier vector array sorting method is selected; If the proportion exceeds the proportion threshold, a hash bitmap and quick sort hybrid sorting method is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

6. An adaptive hybrid sorting system based on hash bitmap and quick sort, characterized by: include: The query module is used to obtain row identifiers that meet the query conditions from the database index and obtain the query result set; a splitting module, configured to split the row identifier into a compression unit identifier and a row number, store the compression unit identifier in an identifier array, and store the row number in a hash bitmap; a sorting module, configured to perform quick sorting on the identifier array using a hybrid sorting method to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; A traversal module, configured to traverse the array sequence and determine a compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; a splicing module, configured to obtain a complete row identifier by splicing the compression unit identifier and the compression unit bitmap, and return a result set based on the complete row identifier for recording; A sorting method selection module is used to select a sorting method based on the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy; In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold; In the case of a memory-first strategy, obtain the ratio between the amount of data in the result set and the total number of records in the table, and select a sorting method based on the ratio and the pre-qualified ratio threshold.

7. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 5.

Images