Data filtering method and apparatus for data query

Abstract

This specification provides a data filtering method and apparatus for data querying. The filtering method is executed through a target device, which acts as a storage node of a database. The method includes determining the target filtering operator involved in the query and obtaining the encoded target data block stored in the target device. Then, using the data characteristics and encoding characteristics of the target data block, a pre-constructed set of operator functions is queried to obtain the target execution function of the target filtering operator. The set of operator functions records the suitable filtering operator execution functions corresponding to various data types and encoding methods. Thus, the target data block can be filtered using the target execution function matched by the query to obtain the filtering result.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of database querying, and more particularly to a data filtering method and apparatus for data querying. Background Technology

[0002] The development of the internet has generated massive data storage demands. This massive data can include tenant or user personal data, privacy data, etc., which is often stored in database clusters in the form of data tables. To efficiently store and manage this data, many mainstream databases currently employ the LSM (Log Structured Merge) storage method, achieving data storage through incremental log writing and multi-level log data merging. Under this architecture, the SSTable (sorted-string table) data structure is generally used for persistent data storage. Based on the SSTable, data is sorted according to the primary key, thus providing ordered data storage, indexing, and query services.

[0003] As digitalization progresses, the amount of data stored in database systems is constantly increasing, and the demand for real-time analysis and querying of large amounts of data is also growing. Databases need to provide high-performance static data scanning capabilities to efficiently support analytical workloads.

[0004] Predicate pushdown is a common method for optimizing the performance of selective scans on large datasets. Specifically, in some data query systems, filtering operators that can reduce the size of intermediate result sets are pushed down to a lower level in the execution chain for computation without affecting the final result set. This reduces the amount of data processed subsequently and accelerates the query process. Some mainstream databases achieve faster SSTable scan speeds by pushing filtering operators down to the storage layer.

[0005] However, data is typically stored after compression and encoding. Therefore, pushing the filtering operators down to the storage layer for initial data filtering scanning necessitates addressing the issue of data encoding and decoding. This problem often becomes a bottleneck in query execution.

[0006] We hope there can be an improved solution that can optimize the filtering and scanning process of encoded data, thereby accelerating data query execution and improving database performance. Summary of the Invention

[0007] In view of the above problems, this specification provides a data filtering method and apparatus for data querying. During data scanning and filtering, the applicable filtering operator execution function for the current data block can be determined, and the data in the encoded state can be filtered using this function, which effectively promotes predicate pushdown and improves data query efficiency.

[0008] According to the first aspect, a data filtering method for data querying is provided, executed through a target device, which acts as a storage node for a database, the method comprising:

[0009] Identify the target filtering operators involved in the query;

[0010] Obtain the encoded target data block stored in the target device;

[0011] Using the data features and encoding features of the target data block, a pre-constructed set of operator functions is queried to obtain the target execution function of the target filtering operator;

[0012] The target data block is filtered using the target execution function to obtain a first filtering result.

[0013] According to one implementation, the operator function set is a function set customized according to the hardware platform and operating environment of the target device.

[0014] In one embodiment of the above implementation, the operator function set is obtained in the following way: obtaining a basic function set, which is a function set generated during the compilation of the database program and applicable to multiple hardware platforms; during the execution of the database program, obtaining an optimized function set according to the runtime environment, which is an additional function set compiled according to the compilation environment during the compilation stage, including optimized execution functions for several functions in the basic function set; adding the optimized function set to the basic function set to obtain the operator function set.

[0015] In one embodiment, the operator function set specifically includes a multi-dimensional function matrix corresponding to the target filtering operator; the multi-dimensional matrix includes several dimensions corresponding to the data features and several dimensions corresponding to the encoding features. In this case, the query process for the operator function set may include: determining the index value of each dimension in the multi-dimensional matrix based on the data features and the encoding features; and determining the corresponding operator function from the function matrix based on the index value of each dimension, as the target execution function.

[0016] In another embodiment, the operator function set specifically includes multiple sub-function matrices corresponding to multiple meta-operations. In this case, the query process for the operator function set may include: determining, from the multiple sub-function matrices, several target sub-function matrices corresponding to several target meta-operations involved in the target filtering operator; querying each target sub-function matrix according to the data features and encoding features to obtain several target sub-functions; and determining the target execution function based on the combination of the several target sub-functions.

[0017] In a specific embodiment, data features may include several of the following: data type, data length, metadata type; encoding features may include encoding method, and auxiliary features corresponding to the encoding method.

[0018] Furthermore, when the encoding method includes bit-aligned data, the auxiliary feature includes alignment bit width;

[0019] When the encoding method includes dictionary-based encoding, the auxiliary features include relevant features of the dictionary.

[0020] In one embodiment, the target data block is a mixed row and column data block; the first filtering result indicates the filtered target data rows; the method further includes: obtaining the target data column in the execution plan corresponding to the query; determining the target data range through the target data rows and target data columns; and decoding the data in the target data range.

[0021] In one example, the first filtering result is in the form of a result bitmap, which indicates whether it belongs to the filtering result with a first value and a second value, respectively.

[0022] According to one embodiment, the target device has a vectorized register that supports Single Instruction Stream Multiple Data Stream (SIMD); the target execution function includes optimization operations for SIMD, the optimization operations including organizing data according to the vector length of the vectorized register, and processing the organized data based on the vectorized register using the instruction set corresponding to the hardware platform.

[0023] In one implementation, the target filtering operator involves a comparison operation; the target data block is encoded based on a dictionary; the execution path of the target execution function includes: searching for the reference word corresponding to the target value of the comparison operation in the dictionary reference table; and comparing the encoded word in the target data block with the reference word in the dictionary reference table.

[0024] In one example of the above implementation, the target filtering operator includes a comparison operator, an IN operator, or a between operator.

[0025] In one example of the above implementation, the target device has a vectorized register that supports Single Instruction Stream Multiple Data Stream (SIMD); the comparison process includes: performing parallel comparison processing using SIMD in the dictionary reference table.

[0026] According to the second aspect, a data filtering device for data querying is provided, deployed in a target device that functions as a storage node for a database, the device comprising:

[0027] The operator determination unit is configured to determine the target filtering operators involved in the query.

[0028] The data block acquisition unit is configured to acquire encoded target data blocks stored in the target device;

[0029] The function query unit is configured to use the data features and encoding features of the target data block to query a pre-built set of operator functions to obtain the target execution function of the target filtering operator;

[0030] The filtering unit is configured to filter the target data block using the target execution function to obtain a first filtering result.

[0031] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0032] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0033] In the embodiments of this specification, based on the research of various encoding methods and data structure characteristics, optimal execution functions corresponding to various encoding features and data features can be pre-designed, thereby designing an operator function set. During the execution of data query, the operator function set is queried according to the current encoding features and data features to match the preferred execution function under the current environment. Especially when the hardware platform supports single instruction stream multiple data stream (SIMD) computation, the execution function can adopt an optimized SIMD execution method for the current hardware platform. Through the above methods, effective data filtering can be achieved before data decoding at the storage layer, reducing the amount of data to be decoded and the amount of intermediate result set data, thereby improving overall query efficiency. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A schematic diagram of the implementation process is shown in one embodiment;

[0036] Figure 2 A flowchart illustrating a data filtering method according to one embodiment is shown;

[0037] Figure 3 This illustrates a specific example of the data filtering process under columnar encoding.

[0038] Figure 4 This illustrates the specific execution process of the filtering function when using dictionary encoding.

[0039] Figure 5 A schematic block diagram of a data filtering apparatus according to one embodiment is shown. Detailed Implementation

[0040] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0041] As mentioned earlier, to accelerate data retrieval, some databases push filtering operators down to the storage layer for initial data filtering scans. This process requires consideration of data encoding and decoding.

[0042] In one related approach, the encoding method for columns is specified when storing data tables. Since the column encoding method is fixed, a filtering scheme corresponding to that encoding method can be selected at the start of the scan, and this filtering scheme can be based on the encoded data. However, in this approach where the encoding method is bound to the metadata schema (which can be specified for each column), the database cannot flexibly choose a more suitable encoding method for storage based on the characteristics of the data, sacrificing some data compression capabilities. With a fixed encoding method, the actual data may trigger bad cases (poor performance situations) of the corresponding encoding algorithm, affecting compression and query performance.

[0043] In another related approach, adaptive encoding is used when storing data tables based on data characteristics. However, the encoded data is not recognized during queries; data scanning and filtering operations are performed on the decoded result set. That is, the data in the stored data blocks is decoded first, followed by scanning and filtering. This approach decouples the data encoding module from the query execution module, but all data in the scanned data blocks needs to be deserialized / decoded. The decoding process can easily become a performance bottleneck during query execution.

[0044] In another related approach, an adaptive encoding method is used to encode and store data blocks based on data characteristics. During data querying, filtering optimization is performed based on data features, and the execution path for decoding and scanning is determined through numerous conditional branches. However, in adaptive encoding where data features vary, this approach is prone to triggering CPU branch prediction failures during execution, disrupting the instruction execution pipeline. Especially in vectorized execution, which is more efficient for large-scale data queries, frequent branch prediction failures can lead to significant performance degradation.

[0045] On another front, to accelerate data querying and filtering, optimization can be considered from the perspective of improving the hardware performance of database storage devices. Limited by Moore's Law, CPU clock speeds have seen limited increases in recent years. Some hardware platforms offer vectorized registers, using Single Instruction Multiple Data (SIMD) operations to improve data processing efficiency. In SIMD, multiple data items are stored in vectorized registers, and a unified control signal is used to control the unified, parallel computation of these multiple data items, thus improving data processing speed.

[0046] Recently, hardware manufacturers have been trying to enhance CPU SIMD capabilities by providing vectorized registers with larger bit widths and better performance, thereby improving computational performance. However, the use of vectorized registers has certain limitations, requiring corresponding modifications to software and algorithms, as well as optimization of the entire computation process to achieve the desired results. In predicate pushdown scenarios, accelerating using SIMD mode requires optimizing specific filtering operators based on the hardware capabilities and instruction sets provided by the hardware manufacturers. However, the hardware configurations and instruction sets provided by different hardware manufacturers are often different, making it difficult for database software to utilize SIMD capabilities for query acceleration or rendering it unsuitable for different platforms, leading to cross-platform compatibility issues.

[0047] In view of the above considerations, in the embodiments of this specification, a data filtering scheme is provided. This scheme can use a filtering execution function suitable for the hardware platform and data characteristics to filter and scan data blocks for adaptively encoded data, thereby accelerating data query.

[0048] Figure 1 The diagram illustrates the implementation process in one embodiment. The overall implementation process can be divided into a compilation phase and an execution phase.

[0049] In a typical database startup process, a target device with storage and computing capabilities compiles the database program code to generate an executable file. Then, running this executable file, i.e., running the database program, activates the target device as a database storage node, enabling it to perform the corresponding storage and query functions.

[0050] Based on the above-mentioned routine process, according to Figure 1 In the illustrated embodiment, during the compilation of the database program, a set of operator functions is also generated, which records the appropriate filtering operator execution functions for various data types and encoding methods. Typically, this set of operator functions can be organized into an operator function matrix to show the correspondence between each operator execution function and various data features and encoding features. The target device stores this operator function matrix and loads it when running the database program for subsequent query execution.

[0051] Specifically, in some embodiments, the database employs an adaptive encoding method, that is, dynamically using different encoding methods to encode the data based on its different characteristics, and then storing the encoded data. Accordingly, the database program includes logic code for adaptive data encoding. The aforementioned operator function matrix can be designed and constructed based on this adaptive encoding logic code.

[0052] Furthermore, to ensure cross-platform compatibility of the database program and leverage the SIMD capabilities of some hardware platforms, in some embodiments, multiple versions of implementation functions are designed using different hardware specifications and vectorized instruction sets provided by different hardware manufacturers for certain filtering operators that can be optimized using SIMD. During the compilation phase, a cross-platform compatible basic operator function matrix is ​​first generated, where the operator execution functions are suitable for various platforms. Additionally, during compilation, the compilation environment is detected, which can indicate some hardware configuration information of the target device. Then, the corresponding version of the optimized operator function is compiled based on the compilation environment. When the database program runs on the target device, the available optimized function versions are determined by the runtime environment and added to the basic operator function matrix, resulting in the final operator function matrix. Thus, the operator function matrix stored in the target device, serving as a storage node, is a set of operator functions adapted to the target device's hardware platform, while the aforementioned database program remains universally applicable to various different platforms.

[0053] After the database program runs on the target device, the target device functions as a storage node in the database system, capable of performing data storage and retrieval functions. In particular, the target device can utilize the aforementioned operator function matrix to accelerate the data retrieval process.

[0054] Specifically, during the query execution phase, the target device (acting as a storage node) retrieves the encoded data block based on the current query request and scans and filters it according to the filtering operators involved in the query. Typically, the query request is an SQL query statement. To accelerate the scanning and filtering, the target device performs query matching within the previously loaded operator function matrix based on the data characteristics and encoding features of the data block, obtaining an optimized execution function for the filtering operators. Thus, the target device can utilize this optimized execution function to filter the encoded data.

[0055] It is understood that the aforementioned operator function matrix is ​​pre-designed based on the encoding logic of the database program. It records the appropriate filtering operator execution functions for various data types and encoding methods. Furthermore, in some embodiments, this operator function matrix is ​​optimized for the target device's hardware platform. Therefore, the optimized execution function obtained by querying this operator function matrix based on the data and encoding characteristics of the current data block will inevitably be an execution function suitable for the hardware characteristics of the target device, the data characteristics of the current data block, and the encoding characteristics. Using such execution functions, data blocks can be scanned and filtered more quickly and accurately, reducing the amount of data that needs to be decoded subsequently, and pushing the filtering operation down to before decoding.

[0056] The implementation method of the data query phase is described in detail below.

[0057] Figure 2 A flowchart illustrating a data filtering method according to one embodiment is shown, wherein the data filtering is used to accelerate or optimize data queries. The method is executed through a target device that runs a database program and acts as a storage node for the database. Specifically, the target device can be any computing unit, server, computer device, etc., with computing, storage, and processing capabilities. Figure 2 As shown, the data filtering method may include the steps described in detail below.

[0058] In step 21, the target filter operators involved in the query are determined. These target filter operators are those determined based on the query request and can be pushed down to the storage layer. The query request can be, for example, in the form of an SQL query statement. In a typical query execution process, filter operators that may be pushed down to the storage layer include comparison operators, IN operators, BETWEEN operators, NULL operators, etc.

[0059] Specifically, in one example, the query request involves a comparison of a field value with a set value. For instance, the target table might be a table containing grades for multiple subjects, and one of the query conditions is that a math score is >= 80. In this case, it can be determined that the target filtering operators involved in the query include comparison operators. In other query conditions, other operators might be involved as target filtering operators.

[0060] In step 22, the encoded target data block stored in the target device is obtained. It can be understood that in a database system, data in a data table is typically divided into data blocks and stored on persistent storage media. For example, in a mainstream database system, the data block size is approximately 2MB, containing multiple key-value pairs (k,v) of data items. Furthermore, in the SStable data format, data items are stored in order of primary key size. The target data block here can be any data block in the target data table to be queried (e.g., the aforementioned grade table). It is important to emphasize that the data items in the target data block are in an encoded state.

[0061] It should be noted that steps 21 and 22 above can be performed in any reasonable order, and no restriction is made here.

[0062] In one embodiment, the database program running on the target device employs an adaptive encoding method to encode and store data. In this case, different data tables and data blocks stored in the storage layer may use different encoding methods. Thus, based on the acquisition of the target data block described above, the data characteristics and encoding characteristics of the target data block can also be extracted.

[0063] Then, in step 23, using the data features and encoding features of the target data block, a pre-constructed set of operator functions is queried to obtain the execution function corresponding to the data features and encoding features, which serves as the target execution function of the target filtering operator.

[0064] According to one implementation, the above-mentioned set of operator functions is compiled during the database program compilation stage (e.g., ...). Figure 1 The set of filter functions (as shown) is generated by recording the appropriate filter operator execution functions for various data types and encoding methods. In one embodiment, this set of operator functions can be organized into an operator function matrix to show the correspondence between each operator execution function and various data features and encoding features.

[0065] According to one implementation, the aforementioned operator function set is a function set customized based on the hardware platform and operating environment of the target device. In this case, the operator function set can be obtained in the following way: During the compilation phase, the target device obtains a basic function set, which is a function set generated during the compilation of the database program and applicable to various hardware platforms. During the execution of the database program on the target device, an optimized function set is obtained according to the operating environment. This optimized function set is a function set compiled according to the compilation environment during the compilation phase, including optimized execution functions for several functions in the basic function set. The target device can then add this optimized function set to the basic function set to obtain the final operator function set. At this point, the operator function set is a function set adapted to the hardware platform and operating environment of the target device.

[0066] It is understandable that the operator function set needs to record the appropriate filtering operator execution functions for various data types and encoding methods. Therefore, this operator function set can be organized and designed based on the encoding logic in the database program, considering the characteristics of various data types and encoding methods. Several operator execution functions are described below as examples.

[0067] In data encoding, bitpacking is commonly used to reduce redundancy. When the number of aligned bits in a data block is the same, the execution path of the corresponding function can include more efficient data unpacking for data with different bit widths, using branchless programming, loop unrolling, and SIMD. In practice, bitpacking is often combined with other encoding methods. Therefore, the function implementation for each step in the above execution path can be specified based on the characteristics of the encoding method (e.g., whether bitpacking has been performed), the corresponding auxiliary characteristics of the encoding method (if bitpacking was performed, the aligned bit width), and the data characteristics. These data characteristics can include, for example, the type of column data, the type of metadata, whether the data is fixed-length or variable-length, etc.

[0068] For some data structures that can help accelerate filtering scans, optimized execution functions can be designed accordingly. For example, dictionary encoding and run-length encoding both involve the use of dictionaries. For these encoding methods, the execution path of the corresponding function can include locating the filtering boundary through binary search of an ordered dictionary, and then directly performing batch operations on dictionary references with better alignment and smaller bit widths to generate the filtering results.

[0069] When the majority of data within a data block is identical, a constant encoding method can be used. This method stores only the data with the highest cardinality as constants, and then stores the other data and their corresponding row indices. For this encoding method, a possible execution path is to filter out most of the data in the data block by comparing the constant data.

[0070] Difference encoding is also a commonly used encoding method, suitable for numeric data types with small value ranges. According to this encoding method, only the smallest value and the difference between each row and the smallest value are stored. For this encoding method, the possible execution path includes calculating the differences in the filtered data, and then comparing the differences between them, thus achieving filtering.

[0071] When performing operations such as comparing dictionary references and interpolation in the above method, SIMD can be further optimized. The smaller bit width of the encoded data allows more data to be filled into the CPU's vector register at once, providing higher computational efficiency.

[0072] Thus, for each filtering operator, different filtering execution functions can be designed based on different encoding methods, different data characteristics, and different optional hardware configurations (such as the bit width of vector registers), and these functions can be organized into a set of operator functions.

[0073] In one embodiment, the operator function set can be represented as an operator function matrix organized for each of the various filtering operators. This operator function matrix is ​​an n-dimensional matrix, where the n dimensions correspond to the various factors on which the design of the various filtering execution functions is based, including data characteristics, encoding characteristics, etc. More specifically, data characteristics may include, for example, data type (Boolean, numeric, string, etc., with numeric types further divided into integer, floating-point, etc.), data length, metadata type, etc.; encoding characteristics may include encoding methods and auxiliary features corresponding to the encoding methods. For example, when the encoding method includes bit-aligned data, the auxiliary feature includes the alignment bit width; when the encoding method includes dictionary-based encoding, the auxiliary feature includes the relevant features of the dictionary. Since different filtering operators have different filtering objectives, the dimensions of the corresponding operator function matrix may differ.

[0074] In a specific example, the filtering operator execution function in an n-dimensional operator function matrix can be represented as A(I1,I2,…,I…). n ) = f, where I1, I2, ..., I n Let f be the index of each of the n dimensions, and f be the specific function expression. For example, I1 corresponds to a fixed-length or variable-length feature in the data features, with a value range of 0 (indicating variable length) or 1 (indicating fixed length), I2 corresponds to the encoding method, and the value is the number of the encoding method, and so on.

[0075] When the operator function set is represented as an operator function matrix, the process of querying the operator function set in step 23 may include determining the target function matrix corresponding to the target filtering operator. For example, when the target filtering operator is a comparison operator, the function matrix corresponding to the comparison operator is determined. Furthermore, based on the data characteristics and encoding characteristics of the target data block, the index values ​​for each dimension of the matrix are determined. For example, in the case of an n-dimensional operator function matrix, that is, based on the specific data characteristics and encoding characteristics of the target data block, each index I1, I2, ..., I... n Assignment. Thus, based on the index values ​​of each dimension, the corresponding operator function is determined from the objective function matrix, serving as the objective execution function.

[0076] In another embodiment, when constructing the operator function set, each filtering operator is broken down into combinations of basic operations, which can be called primitives or meta-operations. For example, a meta-operation may include an operation to read data in a certain way; an operation to store data into a vector register according to a corresponding length; an operation to control the vector register to perform a data processing operation using specific control instructions, and so on. For each meta-operation, the appropriate sub-execution function is determined under various data characteristics and encoding characteristics, resulting in the sub-function matrix corresponding to that meta-operation. Thus, the operator function set may include multiple sub-function matrices corresponding to multiple meta-operations. Since the factors upon which each meta-operation depends may differ, the dimensions of each sub-function matrix may be the same or different.

[0077] In this case, step 23, querying the operator function set, may include first determining several target meta-operations involved in the target filtering operator, that is, decomposing the target filtering operator into a combination of several target meta-operations. Then, from the multiple sub-function matrices, determining several target sub-function matrices corresponding to these target meta-operations. Based on the data characteristics and encoding characteristics of the target data block, querying each target sub-function matrix yields several target sub-functions. The above method can also be used when querying the target sub-function matrix. That is, determining the index value of the target sub-function matrix based on the data characteristics and encoding characteristics, and querying based on the index value. Thus, the target execution function can be obtained based on the combination of the several target sub-functions obtained through the query.

[0078] After the target execution function is determined, in step 24, the target execution function can be used to filter the aforementioned target data block to obtain the first filtering result.

[0079] In one embodiment, the target data table to be queried uses column-oriented encoding, where the target data block is the data in a specific target column of the target data table to be queried. In this case, the first filtering result corresponds to the data in that target column that satisfies the filtering operator. The first filtering result can be decoded, and the decoded data returned to the query engine. Alternatively, if other filtering operators exist, further filtering can be performed based on the first filtering result.

[0080] In another embodiment, the target data table to be queried uses a mixed row and column encoding, where the target data block contains data from multiple rows and columns in the target data table. In this case, the first filtering result indicates the filtered target data rows. Typically, in this situation, it is also necessary to obtain the target data columns from the execution plan corresponding to the current query, and determine the target data range using the target data rows and number of data columns. That is, the target data range consists of data that simultaneously belongs to both the target data columns and the target data rows. Next, the data within the target data range can be decoded, for example, through batch decoding, and the decoded data can be returned to the query engine. Alternatively, if other filtering operators exist, further filtering can be performed based on the target range.

[0081] According to one implementation, the first filtering result can be in the form of a result bitmap, which uses a first value and a second value to indicate whether it belongs to the filtering result. Typically, the result bitmap uses 1 to indicate a hit (belonging to the filtering result) and 0 to indicate a miss (not belonging to the filtering result). For example, in the case of row-column mixed encoding, the number of bits in the result bitmap can correspond to the row number; the bits corresponding to the filtered target data rows are set to 1, and the other bits are 0, thus intuitively indicating the filtering result.

[0082] The following describes a specific example of implementing the above scheme under a specific encoding method.

[0083] Figure 3 This illustrates a specific example of a data filtering process under columnar encoding. It should be noted that this assumes the target device performing the data filtering process has a hardware platform that supports SIMD vectorized registers.

[0084] exist Figure 3 In the process, the target device first determines the target filtering operator, which can be a comparison operator, a between operator, or an IN operator. The between operator can be converted into a comparison operator (e.g., the superposition of two comparisons). The execution process of the IN operator is similar to that of the comparison operator; therefore, the subsequent execution process uses the comparison operator as an example. Then, the data characteristics of the data block are obtained, including the data type (e.g., if it is numeric, it is further determined whether it is an integer or a floating-point number), whether the data is fixed-length, and the encoding characteristics, such as whether bitpacking is performed under columnar encoding, the alignment width, and whether it is constant storage or difference storage. Then, based on this information, it is matched against the target device's operator function set. If no match is found, it means that under the current data characteristics and encoding characteristics, decoding must be performed before filtering. Thus, the branch calculated after decoding is executed. If an execution function is matched, the data block is filtered according to that execution function. Specifically, since the target device supports SIMD, it can be assumed that the matched execution functions include SIMD-optimized operations. Figure 3The above is illustrated as a SIMD computation. Specifically, the optimization operations described above may include organizing data according to the vector length of the vectorized register (e.g., 256 bits), and processing the organized data based on the vectorized register using the instruction set corresponding to the hardware platform. For example, multiple sets of data can be merged into a 256-bit vector (padding for insufficient bits), stored in a register at once, and processed in parallel using a single instruction. Finally, a bitmap is obtained, showing the filtered data items from a specific data block in a defined column.

[0085] In practice, many databases support setting some data items in a table to null values. Accordingly, in this case, it's necessary to combine the results of the NULL operator with those of regular filtering operators. Specifically, an encoded null bitmap (showing which positions correspond to null values) can be read, and based on the bitmap obtained using the execution function, the encoded null bitmap can be filtered to obtain the final result bitmap.

[0086] Figure 4 This illustrates the specific execution process of the filtering function using a dictionary-based encoding method. It is also assumed that the target device performing this data filtering process supports SIMD. Furthermore, it can be understood that the filtering function can be determined through matching the aforementioned set of operator functions.

[0087] like Figure 4 As shown, since the encoding method involves the use of dictionaries, such as dictionary encoding or run-length encoding, the dictionary information can be obtained during the encoding feature extraction stage. It can be an ordered dictionary reference table that records the correspondence between real data and "words" in the dictionary.

[0088] The `between` operator can be converted into a comparison operator. The following describes the execution process of the filtering function for the comparison operator. For the comparison operator, the boundary values ​​for comparison can be determined, and a reference to that boundary value (i.e., the reference word corresponding to the boundary value) can be found in the dictionary reference table. Then, a comparison is performed against that reference word in the dictionary reference table. If the target device has vector registers, this step can be implemented by using SIMD for comparison in the dictionary reference table, such as... Figure 4 As shown. That is, using SIMD, the encoded words of the data items in the data block are determined, and the comparison result is compared with the above-mentioned reference words. Thus, a bitmap of the result can be obtained.

[0089] The IN operator is used to determine whether a data value falls into a certain set S. To do this, references to each value in set S can be looked up in a dictionary, resulting in multiple reference terms. Then, through loop control, this is transformed into comparison operations on the reference terms in the dictionary reference table. Specifically, SIMD can be used to determine the comparison results between the encoded words in the data block and each of the multiple reference terms, thus obtaining the resulting bitmap.

[0090] In other words, for filtering operators involving comparison operations (comparison operators, IN operators, and BETWEEN operators), the reference word corresponding to the target value involved in the comparison can be looked up in the dictionary reference table. Then, the encoded word in the data block is compared with the aforementioned reference word, thereby performing data filtering corresponding to the filtering operator. This comparison operation can be optimized using SIMD. That is, SIMD can be used to perform parallel optimization of the comparison processing in the dictionary reference table, thereby optimizing the comparison process.

[0091] The above description only illustrates the execution process of some filtering operators under certain encoding methods. It should be understood that, based on research into the characteristics of various encoding methods and data structures, optimal execution functions corresponding to various encoding and data features can be designed, thus creating a set of operator functions. During data querying, querying this set of operator functions based on the current encoding and data features allows for the matching of the preferred execution function under the current environment. This enables effective data filtering before data decoding at the storage layer, reducing the amount of data to be decoded and the amount of intermediate result set data, thereby improving overall query efficiency.

[0092] According to another embodiment, with Figure 2 Corresponding to the data filtering method shown, this specification also provides a data filtering device for data querying, which is deployed in a target device. The target device acts as a storage node for the database and can be any computing unit, server, computer device, etc., with computing, storage, and processing capabilities. Figure 5 A schematic block diagram of a data filtering apparatus according to one embodiment is shown. Figure 5 As shown, the data filtering device 500 includes:

[0093] Operator determination unit 51 is configured to determine the target filtering operators involved in the query;

[0094] The data block acquisition unit 52 is configured to acquire the encoded target data block stored in the target device;

[0095] The function query unit 53 is configured to use the data features and encoding features of the target data block to query a pre-built set of operator functions to obtain the target execution function of the target filtering operator;

[0096] The filtering unit 54 is configured to filter the target data block using the target execution function to obtain a first filtering result.

[0097] For details on the execution process of this data filtering device and its various units, please refer to [link to relevant documentation]. Figure 2 The specific implementation process of the described data filtering method will not be repeated here.

[0098] According to other embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform the aforementioned combination. Figure 2 The described method and process.

[0099] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to achieve the foregoing combination. Figure 2 The described method and process.

[0100] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0101] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data filtering method for data querying, executed through a target device, which acts as a storage node for a database, the method comprising: Identify the target filtering operators involved in the query; Obtain the encoded target data block stored in the target device; Using the data features and encoding features of the target data block, a pre-constructed set of operator functions is queried to obtain the target execution function of the target filtering operator; the set of operator functions records the filtering operator execution functions corresponding to various data types and encoding methods; The target data block is filtered using the target execution function to obtain a first filtering result.

2. The method of claim 1, wherein, The operator function set is a set of functions customized according to the hardware platform and operating environment of the target device.

3. The method of claim 2, wherein, The set of operator functions is obtained in the following way: Obtain the basic function set, which is a function set generated during the compilation of the database program and applicable to multiple hardware platforms; During the execution of the database program, an optimized function set is obtained according to the runtime environment. The optimized function set is an additional function set compiled according to the compilation environment during the compilation stage, which includes optimized execution functions for several functions in the basic function set. The optimized function set is added to the basic function set to obtain the operator function set.

4. The method according to claim 1, wherein, The operator function set includes a multi-dimensional function matrix corresponding to the target filtering operator; the multi-dimensionality includes several dimensions corresponding to the data features and several dimensions corresponding to the encoding features; The query pre-constructed set of operator functions yields the target execution function of the target filtering operator, including: Based on the data features and encoding features, determine the index value of each dimension in the multi-dimensional array; Based on the index values ​​of each dimension, the corresponding operator functions are determined from the function matrix and used as the target execution function.

5. The method of claim 1, wherein, The operator function set includes multiple sub-function matrices corresponding to multiple elementary operations; The query pre-constructed set of operator functions yields the target execution function of the target filtering operator, including: From the plurality of sub-function matrices, determine a plurality of target sub-function matrices corresponding to a plurality of target meta-operations involved in the target filtering operator; Based on the data features and encoding features, query each target sub-function matrix to obtain several target sub-functions; The target execution function is determined based on the combination of the aforementioned target sub-functions.

6. The method according to claim 1, wherein, The data characteristics include several of the following: data type, data length, and metadata type; The encoding features include encoding methods and auxiliary features corresponding to the encoding methods.

7. The method according to claim 6, wherein, When the encoding method includes bit-aligned data, the auxiliary feature includes alignment bit width; When the encoding method includes dictionary-based encoding, the auxiliary features include relevant features of the dictionary.

8. The method of claim 1, wherein, The target data block is a mixed row and column data block; the first filtering result indicates the filtered target data rows; the method further includes: Retrieve the target data column from the execution plan corresponding to the query; The target data range is determined by the target data rows and target data columns; Decode the data within the target data range.

9. The method of claim 1, wherein, The first filtering result is in the form of a result bitmap, which indicates whether it belongs to the filtering result with a first value and a second value respectively.

10. The method of claim 1, wherein, The target device has a vectorized register that supports Single Instruction Stream Multiple Data Stream (SIMD); the target execution function includes optimization operations for SIMD, the optimization operations including organizing data according to the vector length of the vectorized register, and processing the organized data based on the vectorized register using the instruction set corresponding to the hardware platform of the target device.

11. The method of claim 1, wherein, The target filtering operator involves a comparison operation; The target data block is encoded based on a dictionary. The execution path of the target execution function includes: Search the dictionary reference table of the dictionary for the reference word corresponding to the target value of the comparison operation; In the dictionary reference table, the encoded words in the target data block are compared with the aforementioned reference words.

12. The method of claim 11, wherein, The target filtering operators include comparison operators, IN operators, or between operators.

13. The method according to claim 11, wherein, The target device has vectorized registers that support Single Instruction Stream Multiple Data Stream (SIMD); the comparison process includes: SIMD is used to perform parallel comparison processing in the dictionary reference table.

14. A data filtering device for data querying, deployed in a target device that functions as a storage node for a database, the device comprising: The operator determination unit is configured to determine the target filtering operators involved in the query. The data block acquisition unit is configured to acquire encoded target data blocks stored in the target device; The function query unit is configured to use the data features and encoding features of the target data block to query a pre-constructed set of operator functions to obtain the target execution function of the target filtering operator; the set of operator functions records the execution functions of the filtering operators corresponding to various data types and various encoding methods; The filtering unit is configured to filter the target data block using the target execution function to obtain a first filtering result.

15. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-13.

Images