A vectorized design and implementation method and apparatus based on a data warehouse

Abstract

This invention discloses a vectorized design and implementation method and apparatus based on a data warehouse. The method includes: in the vectorized execution engine, a volcano model is adopted at the operator level, and an operator tree is used for parsing and execution, processing one tuple at a time; a column-oriented storage structure is used in the vectorized execution engine; and the vectorization of the HashAgg and HashJoin operators is implemented in the vectorized execution engine. The column-oriented storage structure of the vectorized execution engine allows for reading only necessary column data, improving reading efficiency; the batch reading and processing of one tuple at a time significantly reduces the overhead of virtual function calls, allowing the CPU to focus more time on actual computation, thereby improving execution efficiency; the introduction of efficient data structures and algorithms can reduce CPU cycles by an order of magnitude, improving the performance of operations such as scanning, hashing, equality, and memory copying, as well as overall query performance, by several times.

Description

Technical Field

[0001] This invention belongs to the field of big data data warehouses, and relates to a method and apparatus for improving the performance of a big data data warehouse query engine, particularly a vectorized design and implementation method and apparatus based on a data warehouse. Background Technology

[0002] With the development of database hardware and software technologies, classic SQL computing engines have gradually become the performance bottleneck of data warehouses, especially for data warehouse queries involving a large amount of computation.

[0003] In classic SQL calculation engines, expression trees are used for parsing and execution at the expression level. The expression trees corresponding to filter conditions in SQL are as follows: Figure 1 As shown, parsing and execution based on expressiontrees often makes seemingly simple expressions complex to execute. Take the SQL filter condition, `c1<100andc4=10`, as an example. This filter condition is transformed into an expressiontree with 7 nodes in the database. For each row of data in the table, the virtual functions of `eval` in these 7 nodes are triggered once. At the operator level, a volcano model is used, employing an operatortree for parsing and execution. Operators are chained together using iterators. Thus, the SQL is actually compiled into a... Figure 1 The operatortree shown, while the volcano model offers the advantages of simple and clean implementation, involves a long chain of virtual function calls (next) for each row of results. Furthermore, the operatornext virtual function typically contains a chain of virtual function calls (expressioneval). Although the virtual function calls themselves are not particularly expensive, they still take time. While the operations within the virtual functions might be simple, lightweight computations, the numerous virtual function calls required for each row can become extremely costly when dealing with very large datasets.

[0004] In summary, classic SQL calculation engines heavily utilize virtual functions in both expression and operator calculations. Since each row of data needs to undergo this series of operations, the computational framework has a relatively high overhead. Furthermore, the extensive use of virtual functions also limits the compiler's optimization space. Summary of the Invention

[0005] To address the issue of low execution efficiency of classic SQL execution engines when dealing with very large datasets, this invention provides a vectorized design and implementation method and apparatus based on a data warehouse.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In one embodiment of the present invention, a vectorized design and implementation method based on a data warehouse is proposed, the method comprising:

[0008] In the vectorized execution engine, the operator layer adopts the volcano model, which processes a group of tuples at a time and uses the operator tree for parsing and execution.

[0009] The vectorized execution engine employs a columnar storage structure;

[0010] In the vectorized execution engine, the vectorization of the HashAgg operator and the HashJoin operator is implemented.

[0011] Furthermore, by introducing a low-radix global dictionary, two string-based operations are transformed into a single integer-based operation.

[0012] Furthermore, the vectorized implementation of the HashAgg operator is as follows:

[0013] Calculate hash values ​​in batches on the grouping column for the input tuples; calculate hashbucket values ​​in batches based on the calculated hash values;

[0014] Construct a hashtable using open addressing to handle collisions and record the hash table corresponding to each tuple; if memory usage exceeds the memory limit, write the tuple to disk.

[0015] Calculate the aggregation result and update it to the corresponding hashentry;

[0016] Iterate through the hashtable to output the aggregated results. Scan each hashtable, concatenate the aggregated results and aggregated columns into a tuple, and return it.

[0017] If a tuple for the next group exists, reset the hashtable that has been traversed to the end, rebuild the hashtable for the next group, and execute all the above steps until all tuples have been processed.

[0018] Furthermore, the vectorized implementation of the HashJoin operator is as follows:

[0019] Perform batch calculations on the tuples in the Scan internal table to obtain hash values ​​and hashbucket values;

[0020] Build a hashtable from the inner table of Scan;

[0021] Calculate the hash value and hashbucket value in batches for tuples in the Scan table;

[0022] The tuples in the Scan outer table are used to probe the hashtable built by the Scan inner table, and then batch matching is performed. If a match is found, a marker array is used to mark the corresponding position. If no match is found, the next position of the hashbucket is found and the matching continues until a match is successful or the hashbucket chain is completed.

[0023] The rows that match are converted into the corresponding columns based on the marker array.

[0024] In one embodiment of the present invention, a vectorized design and implementation apparatus based on a data warehouse is also proposed, the apparatus comprising:

[0025] The vectorized execution module is used in the vectorized execution engine to adopt the volcano model at the operator level, and to parse and execute by processing a group of tuples at a time using the operator tree.

[0026] The vectorized storage module is used to employ a columnar storage structure in the vectorized execution engine;

[0027] The vectorized operator implementation module is used to implement the vectorization of the HashAgg and HashJoin operators in the vectorized execution engine.

[0028] Furthermore, by introducing a low-radix global dictionary, two string-based operations are transformed into a single integer-based operation.

[0029] Furthermore, the vectorized implementation of the HashAgg operator is as follows:

[0030] Calculate hash values ​​in batches on the grouping column for the input tuples; calculate hashbucket values ​​in batches based on the calculated hash values;

[0031] Construct a hashtable using open addressing to handle collisions and record the hash table corresponding to each tuple; if memory usage exceeds the memory limit, write the tuple to disk.

[0032] Calculate the aggregation result and update it to the corresponding hashentry;

[0033] Iterate through the hashtable to output the aggregated results. Scan each hashtable, concatenate the aggregated results and aggregated columns into a tuple, and return it.

[0034] If a tuple for the next group exists, reset the hashtable that has been traversed to the end, rebuild the hashtable for the next group, and execute all the above steps until all tuples have been processed.

[0035] Furthermore, the vectorized implementation of the HashJoin operator is as follows:

[0036] Perform batch calculations on the tuples in the Scan internal table to obtain hash values ​​and hashbucket values;

[0037] Build a hashtable from the inner table of Scan;

[0038] Calculate the hash value and hashbucket value in batches for tuples in the Scan table;

[0039] The tuples in the Scan outer table are used to probe the hashtable built by the Scan inner table, and then batch matching is performed. If a match is found, a marker array is used to mark the corresponding position. If no match is found, the next position of the hashbucket is found and the matching continues until a match is successful or the hashbucket chain is completed.

[0040] The rows that match are converted into the corresponding columns based on the marker array.

[0041] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned vectorized design and implementation based on a data warehouse.

[0042] In one embodiment of the present invention, a computer-readable storage medium is also proposed, which stores a computer program that performs vectorized design and implementation based on a data warehouse.

[0043] Beneficial effects:

[0044] 1. The vectorized execution engine of this invention adopts a column-oriented storage structure, which can read only the necessary column data, improving reading efficiency; it can use compression algorithms to achieve a better compression ratio; and it can use column sorting or sparse indexes to accelerate data query efficiency. Regarding the effect of vectorized aggregation queries, tests were conducted on the execution results of queries with small and large amounts of data. It was found that the execution time using the column-oriented storage structure is shorter, and the effect is more pronounced with larger data volumes. The execution time using the column-oriented storage structure can even be reduced to 1 / 8 of the execution time using the row-oriented storage structure.

[0045] 2. The vectorized execution engine of this invention still adopts the volcano model, but it processes a group of tuples at a time to achieve batch reading and batch processing, which greatly reduces the overhead of virtual function calls. The CPU can concentrate more time on actual calculations, resulting in higher efficiency.

[0046] 3. By introducing a low-radix global dictionary, this invention transforms two string-based operations into one integer-based operation, which can reduce CPU cycles by an order of magnitude and improve the performance of operations such as scanning, hashing, equality, and memory copying, as well as the overall query performance, by several times, with an overall query performance improvement of over 300%. Attached Figure Description

[0047] Figure 1 This is a schematic diagram illustrating how the classic SQL calculation engine of this invention parses and executes SQL at the expression and operator levels.

[0048] Figure 2 This is an architecture diagram of the vectorized design and implementation method of the data warehouse based on the present invention;

[0049] Figure 3 This is a schematic diagram showing that the storage structure of the vectorized execution engine of this invention has been changed from row storage to column storage;

[0050] Figure 4 This is a schematic diagram of the vectorized implementation of the HashAgg operator according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the vectorized implementation of the HashJoin operator according to an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram illustrating an embodiment of the present invention of converting two string-based groups into one integer-based group;

[0053] Figure 7 This is a schematic diagram of the vectorized design and implementation device based on a data warehouse according to the present invention.

[0054] Figure 8 This is a schematic diagram of the computer device structure of the present invention. Detailed Implementation

[0055] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0056] Those skilled in the art will recognize that embodiments of the present invention can be implemented as an apparatus, device, node, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0057] The embodiments of the present invention propose a vectorized design and implementation method and apparatus based on a data warehouse. The vectorized execution engine allows multiple rows of data to be processed simultaneously using SIMD in each operator. At the same time, vectorized execution processes data in units of tuples, which improves the CPU cache hit rate and makes full use of CPU and cache resources. It can be extended to other application areas, such as data mining and multimedia retrieval.

[0058] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0059] Figure 2 This is an architecture diagram of the vectorized design and implementation method for data warehouses based on this invention. (See diagram for example.) Figure 2 As shown, the method includes:

[0060] 1. Vectorized execution engine

[0061] (1) In the vectorized execution engine, the volcano model is still used at the operator level, and the operator tree is used to parse and execute. The biggest advantage of the volcano model is that it is simple to implement. Each operator only needs to complete its own specific function. The operators are completely decoupled. The compiler only needs to construct the corresponding operator according to the SQL logic and then chain the operators together.

[0062] (2) In terms of reducing framework overhead, amortized overhead is used. If the overhead of generating a row of data, i.e. a tuple, through the operator tree each time is C, the total overhead of the computation framework of the classic SQL execution engine is C*N, where N is the total number of rows involved in the computation. If generating a row of data each time is changed to generating a batch of data, i.e. a group of tuples each time, since the overhead of each call is relatively constant, the total overhead of the computation framework can be reduced to C*N / M, where M is the number of rows in each batch of data. In this way, the overhead of each row is reduced to 1 / M of the original. When M is relatively large, the overhead of the computation framework will not become the system bottleneck.

[0063] 2. Vectorized storage structure

[0064] (1) Row storage

[0065] like Figure 3As shown, each column of each tuple is actually stored contiguously. The advantage of this is that it is easy to add or modify a tuple, but when reading data, unnecessary columns may be read. This is more suitable for OLTP (Online Transaction Processing) scenarios that contain a large number of high-concurrency CRUD transactions.

[0066] (2) Column storage

[0067] like Figure 3 As shown, each column is stored separately, so only the columns that are needed can be read. Writing tuples requires operations on multiple files, which is more suitable for OLAP (Online Analytical Processing) scenarios that involve reading large amounts of data and complex calculations.

[0068] The vectorized execution engine employs a column-oriented storage structure, requiring only the reading of essential columns to improve read efficiency. Utilizing the Snappy compression algorithm (an existing compression algorithm), column-oriented storage achieves a better compression ratio. Data retrieval efficiency is further accelerated through column sorting in the dictionary or sparse indexing. This column-oriented storage structure also greatly facilitates performance optimization for higher-level computations.

[0069] 3. Vectorized Execution Example

[0070] The principles for implementing vectorized operators are: one is to break down complex loop processing into multiple simpler loops as much as possible, so as to process the same type of data quickly in batches; the other is to reduce branches and data dependencies.

[0071] Vectorization of common operators is implemented. For example, in big data warehouse queries, the HashJoin operator is one of the most core SQL operators. Vectorizing the HashJoin operator can improve Join performance by 100% or 30-40%, and the performance improvement is also significant when the Join data volume is large. It also implements intersection, difference, CrossJoin, ScanNode, and window functions. The HashAgg operator uses two columns to group data and performs a count calculation within each group. It includes two steps: first, building a hashtable and calculating the aggregation result on each hash; second, traversing the hashtable and calculating the final aggregation result. Vectorization allows these operations to be processed in batches through simple loops, including the calculation of hash values ​​and hashbucket values, as well as the calculation of aggregation results, which can significantly improve computational efficiency.

[0072] The following section uses the HashAgg (hash aggregation) operator and the HashJoin (hash join) operator as examples to demonstrate the vectorization transformation.

[0073] (1) HashAgg operator

[0074] like Figure 4 As shown, this is a simple query that groups the two columns and then performs a HashAgg operation on each column.

[0075] The specific renovation process is as follows:

[0076] (a) Calculate hash values ​​in batches on the grouping column of the input tuples using a hash function (existing technology); calculate hashbucket values ​​in batches based on the calculated hash values ​​using a hash function (existing technology).

[0077] (b) Construct a hashtable using the Open-Addressing (existing technology) collision handling method to record the hash entry corresponding to the tuple. The hash entry has three fields: hash, value, and next, which correspond to the hash value, tuple, and the hash value of the next tuple, respectively. If the memory usage exceeds the memory limit, the tuple needs to be written to the disk.

[0078] A hashtable is constructed as follows:

[0079] Calculate a hash value based on a tuple;

[0080] Create an array to serve as a hashtable;

[0081] To determine the position of a tuple in an array, perform a bitwise AND operation between the hash value and the maximum index of the array to obtain the index position.

[0082] Check if there is a tuple at this position. If there is no tuple, simply place the tuple at this position.

[0083] If there are tuples, check if the tuples are completely equal. If they are the same, replace the tuple with the current tuple.

[0084] If there is a tuple, but the tuples are not all equal, traverse the linked list at this position and place the tuple at the last position of the linked list.

[0085] (c) Calculate using aggregation function operators such as sum, avg, count, etc., obtain the aggregation result, and update it to the corresponding hashentry.

[0086] (d) Traverse the hashtable to output the aggregation results. Scan each hashtable, concatenate the aggregation results and the aggregation columns into a tuple and return it.

[0087] (e) If there is a tuple in the next group, reset the hashtable that has been traversed to the end, rebuild the hashtable of the next group and perform all the above steps until all tuples have been processed.

[0088] (2) HashJoin operator

[0089] Vectorize common inner joins to... Figure 5 Let's take the HashJoin operator as an example to illustrate its vectorization process. The specific transformation process is as follows:

[0090] (a) Perform batch calculations on the tuples in the inner table of Scan (a table in an inner join) using a hash function (existing technology) to obtain hash values ​​and hashbucket values.

[0091] (b) Construct a hashtable for the Scan internal table.

[0092] (c) Calculate the hash value and hashbucket value in batches for tuples in the Scan table (another table in the inner join).

[0093] (d) Use the tuples in the Scan outer table to probe the hashtable built by the Scan inner table, and then perform batch matching (matching by the key of the Scan inner and outer tables). If a match is found, mark it at the corresponding position using a marker array. If no match is found, find the next position in the hash bucket and continue matching until a match is found or the hash bucket chain matching ends.

[0094] (e) Convert the successfully matched rows into the corresponding columns based on the tag array, for example, find the corresponding pc1 by the key that matches the appearance.

[0095] 4. Efficient data structures and algorithms

[0096] Efficient data structures and algorithms can reduce CPU cycles by an order of magnitude. For example, introducing a low-radix global dictionary can convert string-based operations into integer-based operations. Figure 6 As shown, the operation converts two string-based groups into one integer-based group.

[0097] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0098] Based on the same inventive concept, this invention also proposes a vectorized design and implementation apparatus based on a data warehouse. The implementation of this apparatus can refer to the implementation of the methods described above, and repeated details will not be repeated. The term "module" as used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] Figure 7 This is a schematic diagram of the vectorized design and implementation device based on a data warehouse according to the present invention. Figure 7 As shown, the device includes:

[0100] The vectorized execution module 101 is used in the vectorized execution engine to adopt the volcano model at the operator level, and to use operatortree to parse and execute by processing a group of tuples at a time.

[0101] Vectorized storage module 102 is used to employ a columnar storage structure in the vectorized execution engine.

[0102] The vectorized operator implementation module 103 is used to implement the vectorization of the HashAgg operator and the HashJoin operator in the vectorized execution engine.

[0103] The vectorized implementation of the HashAgg operator is as follows:

[0104] Calculate hash values ​​in batches on the grouping column for the input tuples; calculate hashbucket values ​​in batches based on the calculated hash values;

[0105] Construct a hashtable using open addressing to handle collisions and record the hash table corresponding to each tuple; if memory usage exceeds the memory limit, write the tuple to disk.

[0106] Calculate the aggregation result and update it to the corresponding hashentry;

[0107] Iterate through the hashtable to output the aggregated results. Scan each hashtable, concatenate the aggregated results and aggregated columns into a tuple, and return it.

[0108] If a tuple for the next group exists, reset the hashtable that has been traversed to the end, rebuild the hashtable for the next group, and execute all the above steps until all tuples have been processed.

[0109] The vectorized implementation of the HashJoin operator is as follows:

[0110] Perform batch calculations on the tuples in the Scan internal table to obtain hash values ​​and hashbucket values;

[0111] Build a hashtable from the inner table of Scan;

[0112] Calculate the hash value and hashbucket value in batches for tuples in the Scan table;

[0113] The tuples in the Scan outer table are used to probe the hashtable built by the Scan inner table, and then batch matching is performed. If a match is found, a marker array is used to mark the corresponding position. If no match is found, the next position of the hashbucket is found and the matching continues until a match is successful or the hashbucket chain is completed.

[0114] The rows that match are converted into the corresponding columns based on the marker array.

[0115] By introducing a low-radix global dictionary, two string-based operations are transformed into a single integer-based operation.

[0116] It should be noted that although several modules of the vectorized design and implementation apparatus based on data warehouses have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0117] Based on the aforementioned inventive concept, such as Figure 8 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it implements the aforementioned vectorized design and implementation method based on a data warehouse.

[0118] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned vectorized design and implementation based on a data warehouse.

[0119] This invention proposes a vectorized design and implementation method and apparatus based on a data warehouse. The vectorized execution engine adopts a column-oriented storage structure, which allows reading only necessary columns, improving read efficiency. Compression algorithms can be used to achieve better compression ratios. Column sorting or sparse indexes can be used to accelerate data query efficiency. Tests on the effectiveness of vectorized aggregation queries, using small and large datasets, show that the column-oriented storage structure results in shorter execution time, with the effect becoming more pronounced as the data volume increases. The execution time of the column-oriented storage structure can even be reduced to 1 / 8 of that of the row-oriented storage structure. The vectorized execution engine still uses the volcano model, but processes tuples one at a time, achieving batch reading and batch processing, significantly reducing virtual function call overhead. The CPU can then focus more time on actual computation, resulting in higher efficiency. By introducing a low-cardinality global dictionary, two string-based operations are converted into one integer-based operation, reducing CPU cycles by an order of magnitude. This improves the performance of operations such as scanning, hashing, equality, and memory copying, as well as overall query performance, by several times, with an overall query performance improvement of over 300%.

[0120] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0121] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A vectorized design and implementation method based on a data warehouse, characterized in that, The method includes: In the vectorized execution engine, the operator layer adopts the volcano model, which processes a group of tuples at a time and uses an operator tree to parse and execute them. The vectorized execution engine employs a columnar storage structure; In the vectorized execution engine, the vectorization of the HashAgg operator and the HashJoin operator is implemented; The vectorized implementation of the HashAgg operator is as follows: Calculate hash values ​​in batches on the grouping column for the input tuples; calculate hashbucket values ​​in batches based on the calculated hash values; A hash table is constructed using the open addressing method for collision resolution, and the hash entry corresponding to the tuple is recorded. If the memory usage exceeds the memory limit, the tuple is written to disk. Calculate the aggregation result and update the corresponding hash entry; Iterate through the hash table to output the aggregated results. Scan each hash entry, concatenate the aggregated results and aggregated columns into a tuple, and return it. If there is a tuple in the next group, reset the hash table that has been traversed to the end, rebuild the hash table of the next group and execute all the above steps until all tuples have been processed. The vectorized implementation of the HashJoin operator is as follows: Perform batch calculations on the tuples in the Scan internal table to obtain hash values ​​and hash bucket values; Build a hash table for the internal table of Scan; Calculate hash values ​​and hash bucket values ​​in batches for tuples in the Scan table; The tuples in the Scan outer table are used to probe the hash table built by the Scan inner table, and then batch matching is performed. If a match is found, a marker array is used to mark the corresponding position. If no match is found, the next position in the hash bucket is found and the matching continues until a match is successful or the hash bucket chain matching ends. The rows that match are converted into the corresponding columns based on the marker array.

2. The vectorized design and implementation method based on a data warehouse according to claim 1, characterized in that, By introducing a low-radix global dictionary, two string-based operations are transformed into a single integer-based operation.

3. A vectorized design and implementation device based on a data warehouse, characterized in that, The device includes: The vectorized execution module is used in the vectorized execution engine to adopt the volcano model at the operator level, and to parse and execute by processing a group of tuples at a time using the operator tree; The vectorized storage module is used to employ a columnar storage structure in the vectorized execution engine; The vectorized operator implementation module is used to implement the vectorization of the HashAgg operator and the HashJoin operator in the vectorized execution engine; The vectorized implementation of the HashAgg operator is as follows: Calculate hash values ​​in batches on the grouping column for the input tuples; calculate hashbucket values ​​in batches based on the calculated hash values; A hash table is constructed using the open addressing method for collision resolution, and the hash entry corresponding to the tuple is recorded. If the memory usage exceeds the memory limit, the tuple is written to disk. Calculate the aggregation result and update the corresponding hash entry; Iterate through the hash table to output the aggregated results. Scan each hash entry, concatenate the aggregated results and aggregated columns into a tuple, and return it. If there is a tuple in the next group, reset the hash table that has been traversed to the end, rebuild the hash table of the next group and execute all the above steps until all tuples have been processed. The vectorized implementation of the HashJoin operator is as follows: Perform batch calculations on the tuples in the Scan internal table to obtain hash values ​​and hash bucket values; Build a hash table for the internal table of Scan; Calculate hash values ​​and hash bucket values ​​in batches for tuples in the Scan table; The tuples in the Scan outer table are used to probe the hash table built by the Scan inner table, and then batch matching is performed. If a match is found, a marker array is used to mark the corresponding position. If no match is found, the next position in the hash bucket is found and the matching continues until a match is successful or the hash bucket chain matching ends. The rows that match are converted into the corresponding columns based on the marker array.

4. The vectorized design and implementation device based on a data warehouse according to claim 3, characterized in that, By introducing a low-radix global dictionary, two string-based operations are transformed into a single integer-based operation.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 1-2.

Images