Self-adaptive hybrid sorting method and system based on Hash bitmap and rapid sorting

Abstract

The invention belongs to the field of databases, and particularly discloses a self-adaptive hybrid sorting method and system based on a Hash bitmap and rapid sorting, and the method comprises the steps: obtaining a row identifier meeting a query condition from a database index, and obtaining a query result set; splitting the line identifier into a compression unit identifier and a line number, storing the compression unit identifier into an identifier array, and storing the line number into a hash bitmap; quickly sorting the identifier array by using a hybrid sorting method to obtain an array sequence of the compression unit identifiers; the hybrid sorting method is realized based on a Hash bitmap and rapid sorting; traversing the array sequence, and determining a compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; and splicing according to the compression unit identifier and the compression unit bitmap to obtain a complete row identifier, and returning a result set for recording based on the complete row identifier. According to the invention, the query performance and efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the field of databases, and more specifically, to an adaptive hybrid sorting method and system based on hash bitmap and quick sorting. Background Art

[0002] In database systems, indexes are an important means of improving query performance. When a query returns a large number of result sets but does not reach the scale required to process all table data, the optimizer usually weighs between index scanning and sequential scanning. Since index scanning introduces a large number of random IO operations, the cost is higher than that of sequential scanning, so the optimizer usually chooses the bitmap scan method to reduce the cost of the execution plan.

[0003] However, the traditional column-stored table bitmap scan uses an ordered CTID vector array instead of a bitmap. Although this solves the problem of memory waste when the result set is sparse, it also introduces technical problems such as high sorting time complexity, high memory usage, and large external sorting overhead. These problems are particularly prominent in large-scale OLAP query scenarios, resulting in poor query performance and efficiency of the database, which is difficult to further improve.

[0004] Therefore, an efficient column-stored table bitmap scanning optimization solution is urgently needed to improve the query performance and efficiency of the database. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of the present application is to provide an adaptive hybrid sorting method and system based on hash bitmap and quick sort, aiming to solve the problem of poor query performance of database systems in the prior art.

[0006] To achieve the above objectives, in a first aspect, the present application provides an adaptive hybrid sorting method based on hash bitmap and quick sort, which is applied to a column storage table of a database, comprising: Obtain row identifiers that meet the query conditions from the database index to obtain the query result set; Splitting the row identifier into a compression unit identifier and a row number, storing the compression unit identifier in an identifier array, and storing the row number in a hash bitmap; Using a hybrid sorting method to quickly sort the identifier array to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; Traversing the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; A complete row identifier is obtained by concatenating the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

[0007] Optionally, the concatenation method of the complete row identifier includes: Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap; A complete row identifier is obtained by concatenating the compression unit identifier with the effective row number, so as to read and return data records in a sequence based on the complete row identifier.

[0008] Optionally, it also includes: Select a sorting method according to the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy; In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold; In the case of the memory priority strategy, the ratio between the amount of data in the result set and the total number of table records is obtained, and the sorting method is selected based on the ratio and the pre-qualification ratio threshold.

[0009] Optionally, the identifier array stores only the deduplicated compression unit identifiers in the result set; Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap; Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the condition is filtered.

[0010] Optionally, the method of storing the row number in the hash bitmap includes: Using the compression unit identifier as a key, the corresponding row number of the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

[0011] Optionally, the method for determining the sorting method includes: If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed a preset threshold, a row identifier vector array sorting method is selected; If the proportion exceeds the proportion threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

[0012] Optionally, the row identifier vector array sorting method includes: Store the row identifiers obtained from the query into the row identifier array; Sort the row identifier array to obtain an array sequence; Traverse the sorted array sequence, access the compression unit corresponding to each row identifier in sequence, and read the corresponding data record from the compression unit according to the row number.

[0013] In a second aspect, the present application also provides an adaptive hybrid sorting system based on hash bitmap and quick sort, comprising: The query module is used to obtain row identifiers that meet the query conditions from the database index and obtain the query result set; A splitting module, used for splitting the row identifier into a compression unit identifier and a row number, storing the compression unit identifier in an identifier array, and storing the row number in a hash bitmap; A sorting module, used for quickly sorting the identifier array using a hybrid sorting method to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; A traversal module, used for traversing the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; The splicing module is used to obtain a complete row identifier according to the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

[0014] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0016] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0017] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0018] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: (1) This application splits the row identifier into a compression unit identifier and a row number, and stores them in an array and a hash bitmap respectively, effectively reducing the size of the data to be sorted. The compression unit identifier is a high-frequency access primary key, and is stored independently for easy sorting, while the row number is stored in a compact binary form through a hash bitmap, saving continuous memory space. The split storage method not only reduces memory usage, but also avoids the duplication of redundant data in full sorting, thereby greatly reducing the sorting time, thereby improving the query performance and efficiency of the database.

[0019] (2) This application further optimizes the sorting performance by introducing a hybrid sorting method, combining the advantages of hash bitmap and quick sort. The hash bitmap pre-groups the row numbers associated with the compression unit identifiers, avoiding the frequent cross-compression unit identifier comparison operations in traditional full sorting, and significantly reducing the sorting complexity. Quick sort performs local sorting on the compression unit identifiers after bucketing, taking into account both global orderliness and local efficiency. The hybrid sorting method not only adapts to sparse or dense data distribution characteristics, but also maintains stable performance under extreme data distribution, thereby further improving the sorting efficiency.

[0020] (3) This application can select the appropriate sorting method according to performance and memory requirements, so that the system can flexibly adjust resource usage and performance optimization when processing data of different sizes. The identifier array only stores the deduplicated compression unit identifiers, which effectively reduces memory usage and improves data processing efficiency.

[0021] (4) This application dynamically selects a sorting method based on the amount of data in the query result set and the proportion of the total number of table records, thereby improving the algorithm's adaptability and enabling it to maintain good performance in different usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is one of the flow charts of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application; Figure 2 This is the second flow chart of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application; Figure 3 This is the third flow chart of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application; Figure 4 This is a flowchart of the traditional CTID array qsort sorting; Figure 5 This is the fourth flow chart of the adaptive hybrid sorting method based on hash bitmap and quick sort provided in the embodiment of the present application; Figure 6It is a structural schematic diagram of an adaptive hybrid sorting system based on hash bitmap and quick sort provided in an embodiment of the present application; Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0025] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0027] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0028] First, the terms of the embodiments of the present application are explained in detail.

[0029] Index: A structure that sorts the values ​​of one or more columns in a database table. Using an index can quickly access specific information in a database table and is a means of optimizing database performance.

[0030] Row storage: Also known as row storage or horizontal storage, it is a traditional database storage method. In this method, data is organized according to row-based logical storage units. Each row contains several fields or columns, which are arranged in a certain order.

[0031] Column storage: also known as column storage or vertical storage, is an emerging database storage method. In this method, data is organized based on columns as logical storage units. Each column contains several fields or values, which are compressed and stored in CUs. A column can have multiple CUs.

[0032] CU (Compression Unit): Compression unit is the smallest storage unit of column-stored data. Some column-stored databases call it a segment. Usually a CU is formed by compressing tens of thousands of records of a single field.

[0033] cuid: CU number, that is, compression unit identifier.

[0034] CTID: Row identifier, a system column provided by PostgreSQL, the field type is TID (TupleIdentifier), which is a unique identifier for each record, used to identify the physical storage location of each row record in the table. The ctid of a row-stored table consists of two parts: the block number (4 bytes) and the row number (2 bytes). For example: (0, 10) means the 10th row in physical block 0. The ctid of a column-stored table consists of cuid (4 bytes) and row number (2 bytes).

[0035] qsort (Quick Sort): A C language standard library function based on the quick sort algorithm. The average time complexity is O(NlogN). In some specific cases (such as an already ordered array), the time complexity will degenerate to O(N^2).

[0036] Quick sort: Adopting the divide-and-conquer idea, an arbitrary pivot element is selected from an unordered sequence, and the sequence to be sorted is divided into two parts using the pivot. The elements in the front part are all less than or equal to the pivot element, and the elements in the back part are all greater than or equal to the pivot element. Then, a recursive method is used to repeat the above operations on the front and back parts respectively until the unordered sequence is arranged into an ordered sequence.

[0037] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0038] Reference Figure 1 , an adaptive hybrid sorting method based on hash bitmap and quick sort, applied to a column storage table of a database, including: The present application provides an adaptive hybrid sorting method based on hash bitmap and quick sort, applied to a column storage table of a database, including: S101. Obtain row identifiers that meet the query conditions from the database index to obtain a query result set; S102. Split the row identifier into a compression unit identifier and a row number, store the compression unit identifier in an identifier array, and store the row number in a hash bitmap; S103. Use a hybrid sorting method to quickly sort the identifier array to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; S104. Traverse the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; S105. Obtain a complete row identifier according to the compression unit identifier and the compression unit bitmap, and return a result set based on the complete row identifier for recording.

[0039] First of all, it should be noted that the embodiment of the present application is applied to the bitmap scan operator optimization of the column storage table.

[0040] In step S101, row identifiers that meet the query conditions are obtained from the database index to obtain the query result set. The core of this step is to quickly locate the rows that meet the query conditions through the index. As a key data structure for efficient query in the database, the index can significantly reduce the cost of full table scanning. After obtaining the row identifiers through the index, these identifiers represent the physical location of the data rows that meet the conditions in the database, providing a basis for subsequent processing.

[0041] It should be noted that the query conditions in the embodiments of the present application are filtering rules set by the user or program when querying data, which are used to find data that meets the requirements. Common query conditions include: Equal to a certain value: For example, filter out a column whose value is equal to a specific data.

[0042] Within a certain range: For example, filter out values ​​in a column that are greater than or less than a certain number, or within a certain interval.

[0043] Fuzzy matching: For example, filter out values ​​in a column that match a certain pattern, such as starting with a certain word.

[0044] Combination conditions: For example, multiple conditions are met at the same time, or one of the conditions is met.

[0045] Null value check: For example, filter out the value of a column to be null or not null.

[0046] In a certain collection: For example, filter out values ​​of a column that belong to a specific collection.

[0047] In the embodiment of the present application, the query condition is first used to find the row identifiers that meet the condition from the index of the database. Then, these row identifiers are split into two parts: compression unit identifier and row number, which are stored in different data structures respectively. Then, through sorting and bitmap processing, a complete result set is finally generated and returned to the user. In other words, the query condition is the rule used to filter data and is the basis for obtaining the result set.

[0048] In step S102, the row identifier is split into a compression unit identifier and a row number, and the compression unit identifier is stored in an identifier array, and the row number is stored in a hash bitmap.

[0049] The row identifier usually contains two parts of information: the compression unit identifier is used to identify the compression unit where the data is located, and the row number is used to identify the specific row in the compression unit. Through split storage, the compression unit identifier is centrally stored in an array to facilitate subsequent sorting operations, while the row number is stored in a compact binary form through a hash bitmap, saving memory space and supporting fast positioning.

[0050] Reference Figure 2 , Figure 2 It is a schematic diagram of the data structure of the hybrid sorting method HBQSAH sorting in the embodiment of the present application.

[0051] In step S103, the identifier array is quickly sorted using a hybrid sorting method to obtain an array sequence of compression unit identifiers.

[0052] The hybrid sorting method combines the advantages of hash bitmap and quick sort. First, the row numbers associated with the compression unit identifiers are pre-grouped through the hash bitmap to avoid frequent cross-compression unit identifier comparison operations in traditional full sorting, thereby reducing the sorting complexity. Then, the compression unit identifiers after bucketing are quickly sorted, taking into account global orderliness and local efficiency, ensuring that the sorting performance remains stable under different data distributions.

[0053] In step S104, the sorted array sequence is traversed to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap. The hash bitmap is used to quickly locate the row number corresponding to each compression unit identifier, thereby avoiding the overhead of parsing the compression unit metadata row by row and further shortening the data extraction time.

[0054] In step S105, a complete row identifier is obtained by concatenating the compression unit identifier and the compression unit bitmap, and a result set is returned based on the complete row identifier for recording.

[0055] The splicing process recombines the compression unit identifier with the row number in the hash bitmap to generate a complete row identifier, ensuring the accuracy and completeness of the result set. By splicing on demand and returning results in a streaming manner, query results are gradually returned to avoid waiting delays before the full sort is completed, while reducing network or memory bandwidth pressure, significantly improving query response speed.

[0056] Optionally, the concatenation method of the complete row identifier includes: Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap; A complete row identifier is obtained by concatenating the compression unit identifier with the effective row number, so as to read and return data records in a sequence based on the complete row identifier.

[0057] Specifically, firstly, the hash value of the compression unit identifier is calculated, and the corresponding compression unit bitmap is extracted from the hash bitmap. This step quickly locates the target bitmap through hash mapping, providing a basis for subsequent parsing.

[0058] Then the compression unit bitmap is parsed, and the row numbers marked as valid are traversed bit by bit to identify them, and a list of valid row numbers is generated. This process significantly reduces storage and computational overhead through the efficient representation method of the bitmap. Then, the compression unit identifier is concatenated with the valid row number to generate a complete row identifier. This step ensures the accurate location of data records through the construction of a unique identifier.

[0059] Finally, based on the order of the complete row identifiers, the corresponding data records are read and returned from the data storage system.

[0060] For example, if cuid=100, the bitmap is 000101, and each bit is checked from back to front. If =1, it is a valid bit, and its position is the corresponding row number, which is 1 and 3. The ctid of the spliced ​​output is (100, 1) and (100, 3).

[0061] This process ensures data consistency and continuity through sequential reading, while improving retrieval efficiency. The entire solution uses key technologies such as hash bitmap, bitmap parsing, and identifier splicing to achieve efficient and flexible data record reading, which is suitable for large-scale data storage and retrieval scenarios.

[0062] Reference Figure 3 , Figure 3 It is a complete flow chart of the application embodiment, including the following steps: First, search for the ctid that meets the conditions in the index, and split the found ctid into two parts: the cuid array and the hash bitmap. Next, determine whether the cuid array is empty: if it is not empty, get the next crtid that meets the conditions, and sort the cuid array with qrsort; if it is empty, end the process directly. After the sorting is completed, traverse the cuid array, get the next cuid, and determine whether it is empty: if it is not empty, load the CU data according to the cuid, and traverse the bitmap corresponding to the cuid to get the next valid bit of the bitmap; if it is empty, end the process. In the process of traversing the bitmap, splice the ctid according to the cuid and the bitmap, and return the record.

[0063] Optionally, it also includes: Select a sorting method according to the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy; In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold; In the case of the memory priority strategy, the ratio between the amount of data in the result set and the total number of table records is obtained, and the sorting method is selected based on the ratio and the pre-qualification ratio threshold.

[0064] Furthermore, the method for determining the sorting method includes: If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed a preset threshold, a row identifier vector array sorting method is selected; If the proportion exceeds the proportion threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

[0065] Specifically, the embodiment of the present application can adaptively select whether the operator uses traditional CTID vector array sorting or HBQSAH sorting based on the ratio of the data volume of the estimated result set to the total number of records in the table and the sorting optimization strategy (performance priority, memory priority).

[0066] If the performance priority strategy is used, if the estimated data volume of the result set exceeds the specified threshold (the default is 1000), HBQSAH sorting is used, otherwise CTID vector array sorting is used; If the memory priority strategy is used, if the estimated amount of data in the result set accounts for more than 2% of the total number of records in the table, HBQSAH sorting is used; otherwise, CTID vector array sorting is used.

[0067] The following is a specific implementation case. The detailed implementation steps are as follows: A column-stored table of an OLAP application has 200 fields, a total of 50 million records, and a data volume of 6.9 GB, of which 833 CUs have 60,000 records and 1 CU has 20,000 records.

[0068] 1) Set the performance priority policy, with the default threshold of 1000; The result set is 1000, so choose traditional qsort. The result set is 10,000, select HBQSAH sorting; If the result set is 100,000, select HBQSAH sort.

[0069] 2) Set the performance priority policy with a threshold of 20000; The result set is 10,000, so choose traditional qsort. If the result set is 100,000, select HBQSAH sort.

[0070] 3) Set the memory priority policy and set the threshold to 2% of the total number of records in the table; The result set is 10,000, so choose traditional qsort. If the result set is 100,000, choose traditional qsort. If the result set is over 1 million, select HBQSAH sort.

[0071] Optionally, the identifier array stores only the deduplicated compression unit identifiers in the result set; Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap; Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the condition is filtered.

[0072] Optionally, the method of storing the row number in the hash bitmap includes: Using the compression unit identifier as a key, the corresponding row number of the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

[0073] Specifically, the sorting method of the embodiment of the present application replaces the CTID vector array with a cuid array and a bitmap hash table. The cuid array only stores the cuids of the result set without duplication. Each member of the bitmap hash table consists of cuid, bitmap size and CU bitmap. Figure 3cuid is used as the key of the hash table to quickly query the bitmap. The CU bitmap marks the row number of the record that meets the conditions in the CU. Each bit of the bitmap represents a row of records in the CU. A value of 1 indicates that the record meets the conditions, and a value of 0 indicates that the record is filtered out.

[0074] Optionally, the row identifier vector array sorting method includes: Store the row identifiers obtained from the query into the row identifier array; Sort the row identifier array to obtain an array sequence; Traverse the sorted array sequence, access the compression unit corresponding to each row identifier in sequence, and read the corresponding data record from the compression unit according to the row number.

[0075] Specifically, refer to Figure 4 , Figure 4 This is a schematic diagram of the traditional CTID array qsort sorting process, which includes the following steps: First, search for the ctid that meets the conditions in the index and put the found ctid into the CTID array. Next, determine whether the CTID array is empty: if the array is not empty, get the next ctid that meets the conditions and sort the CTID array with QSort; if the array is empty, end the process directly. After the sorting is completed, traverse the CTID array, get the next ctid, and determine whether it is empty: if it is not empty, return the record according to the ctid; if it is empty, end the process. The whole process returns the records that meet the conditions through step-by-step searching, sorting, and traversal.

[0076] Further, refer to Figure 5 , Figure 5 : is a schematic diagram of the Index Ctid Scan operator adaptive hybrid sorting process of an embodiment of the present application, including the following steps: First, choose whether to use the sorting algorithm based on the estimated number of result sets and the priority strategy. If you choose to use sorting, perform an index scan, put all CTIDs that meet the conditions into an array, then sort the CTID array, and finally get data from the CU based on the sorted CTID array. If you do not use sorting, perform an index scan, split all CTIDs that meet the conditions and put them into a hash bitmap, then sort the cuid array, and finally get data from the CU based on the hash bitmap.

[0077] Based on the estimated ratio of the data volume of the result set to the total number of records in the table and the sorting optimization strategy (performance first, memory first), the operator is adaptively selected to use traditional CTID vector array sorting or HBQSAH sorting.

[0078] If you choose the traditional CTID vector array sort, the original process is followed and the result set is returned; If HBQSAH sorting is selected, first scan the CTIDs of all rows that meet the conditions in the index, split them into cuid and row numbers, and put them into the cuid array and bitmap hash respectively. The cuid array is deduplicated. When this step is completed, the row numbers in the bitmap are in order, and you only need to sort the cuid array; Sort the cuid array qsort; Load CU in the order of cuid array, generate ctid according to the valid bits in bitmap and cuid, and get the corresponding records in CU in sequence.

[0079] Reference Figure 6 , the present application also provides an adaptive hybrid sorting system based on hash bitmap and quick sort, including: A query module 610 is used to obtain row identifiers that meet the query conditions from the database index to obtain a query result set; A splitting module 620, configured to split the row identifier into a compression unit identifier and a row number, store the compression unit identifier in an identifier array, and store the row number in a hash bitmap; A sorting module 630 is used to quickly sort the identifier array using a hybrid sorting method to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; A traversal module 640 is used to traverse the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; The splicing module 650 is used to obtain a complete row identifier according to the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

[0080] Optionally, the concatenation method of the complete row identifier includes: Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap; A complete row identifier is obtained by concatenating the compression unit identifier with the effective row number, so as to read and return data records in a sequence based on the complete row identifier.

[0081] Optionally, a sorting method selection module is further included, for: Select a sorting method according to the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy; In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold; In the case of the memory priority strategy, the ratio between the amount of data in the result set and the total number of table records is obtained, and the sorting method is selected based on the ratio and the pre-qualification ratio threshold.

[0082] Optionally, the identifier array stores only the deduplicated compression unit identifiers in the result set; Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap; Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the condition is filtered.

[0083] Optionally, the method of storing the row number in the hash bitmap includes: Using the compression unit identifier as a key, the corresponding row number of the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

[0084] Optionally, the method for determining the sorting method includes: If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed a preset threshold, a row identifier vector array sorting method is selected; If the proportion exceeds the proportion threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

[0085] Optionally, the row identifier vector array sorting method includes: Store the row identifiers obtained from the query into the row identifier array; Sort the row identifier array to obtain an array sequence; Traverse the sorted array sequence, access the compression unit corresponding to each row identifier in sequence, and read the corresponding data record from the compression unit according to the row number.

[0086] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0087] Reference Figure 7Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor (Processor) 710, a communication interface (Communications Interface) 720, a memory (Memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730 to execute the method in the above embodiment.

[0088] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0089] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0090] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0091] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0092] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0093] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0094] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0095] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An adaptive hybrid sorting method based on hash bitmap and quick sort, characterized in that: Column-stored tables applied to databases include: Obtain row identifiers that meet the query conditions from the database index to obtain the query result set; Splitting the row identifier into a compression unit identifier and a row number, storing the compression unit identifier in an identifier array, and storing the row number in a hash bitmap; Using a hybrid sorting method to quickly sort the identifier array to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; Traversing the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; A complete row identifier is obtained by concatenating the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

2. The adaptive hybrid sorting method based on hash bitmap and quick sort according to claim 1, characterized in that: The concatenation methods of complete row identifiers include: Extracting a corresponding compression unit bitmap from the hash bitmap according to the compression unit identifier, and parsing a valid row number of the compression unit bitmap; A complete row identifier is obtained by concatenating the compression unit identifier with the effective row number, so as to read and return data records in a sequence based on the complete row identifier.

3. The adaptive hybrid sorting method based on hash bitmap and quick sort according to claim 1, characterized in that: Also includes: Select a sorting method according to the result set and a priority strategy, wherein the priority strategy includes a performance priority strategy and a memory priority strategy; In the case of a performance-first strategy, the sorting method is selected based on the amount of data in the result set and the preset threshold; In the case of the memory priority strategy, the ratio between the amount of data in the result set and the total number of table records is obtained, and the sorting method is selected based on the ratio and the pre-qualification ratio threshold.

4. The adaptive hybrid sorting method based on hash bitmap and quick sorting according to claim 1, characterized in that: The identifier array stores only the deduplicated compression unit identifiers in the result set; Each member in the hash bitmap includes a compression unit identifier, a bitmap size, and a compression unit bitmap, wherein the compression unit identifier is a key of the hash bitmap, wherein the key is used to quickly locate the corresponding compression unit bitmap; Each bit in the compression unit bitmap indicates whether a row of records in the compression unit meets the query condition. A value of 1 indicates that the condition is met, and a value of 0 indicates that the condition is filtered.

5. The adaptive hybrid sorting method based on hash bitmap and quick sorting according to claim 1, characterized in that: Methods for storing row numbers in a hash bitmap include: Using the compression unit identifier as a key, the corresponding row number of the compression unit identifier is mapped to the binary bit of the compression unit bitmap, where 1 represents a valid row and 0 represents an invalid row.

6. The adaptive hybrid sorting method based on hash bitmap and quick sorting according to claim 3, characterized in that: The method for determining the sorting method includes: If the data volume of the result set exceeds a preset threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the data volume of the result set does not exceed a preset threshold, a row identifier vector array sorting method is selected; If the proportion exceeds the proportion threshold, a mixed sorting method of hash bitmap and quick sort is selected; if the proportion does not exceed the proportion threshold, a row identifier vector array sorting method is selected.

7. An adaptive hybrid sorting system based on hash bitmap and quick sort, characterized in that: include: The query module is used to obtain row identifiers that meet the query conditions from the database index and obtain the query result set; A splitting module, used for splitting the row identifier into a compression unit identifier and a row number, storing the compression unit identifier in an identifier array, and storing the row number in a hash bitmap; A sorting module, used for quickly sorting the identifier array using a hybrid sorting method to obtain an array sequence of compression unit identifiers; the hybrid sorting method is implemented based on a hash bitmap and quick sorting; A traversal module, used for traversing the array sequence to determine the compression unit bitmap corresponding to each compression unit identifier in the hash bitmap; The splicing module is used to obtain a complete row identifier according to the compression unit identifier and the compression unit bitmap, so as to return a result set based on the complete row identifier for recording.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images