Efficient scanning through full bitmap index of columnar storage format

Abstract

A system and method for executing a query in a data analytics storage engine is provided. The method includes receiving a query to locate target data in a data analytics storage engine, the data analytics storage engine including data rows divided into one or more partitions of data, the one or more partitions having data columns corresponding to the data rows, and bitmap data embedded in the one or more partitions, wherein the bitmap data is associated with the data columns; and locating the target data using the bitmap data in the one or more partitions.

Description

Technical Field

[0001] This disclosure generally relates to columnar storage indexes, and more specifically, to bitmap indexes in large-scale distributed data analytics storage engines. Background Technology

[0002] Big data technologies have enabled the processing of massive amounts of data, including real-time data, making it possible to make real-time business decisions through large-scale distributed big data analytics. Open-source columnar storage formats for distributed data query processing engines (e.g., Apache Parquet, Apache ORC) have been developed to allow efficient analysis of underlying data through various SQL query processing. However, these columnar storage formats typically have a significant drawback: they lack a proper, comprehensive embedded index structure that provides efficient predicate pushdown to scan only data that matches the filter predicate. Summary of the Invention

[0003] This disclosure provides a method for executing SQL queries in a data analysis storage engine. The method includes receiving a query to scan matching related data in a columnar storage, including: data rows divided into one or more file segments, each file segment being stored in a columnar manner, wherein a bitmap index associated with its corresponding column is embedded together with the column data of each file segment. During query processing, the bitmap index is used to effectively skip data portions that do not conform to the filtering predicate, thereby achieving efficient scanning and significantly reducing I / O costs.

[0004] Furthermore, embodiments of this disclosure provide a data analysis storage engine. The data analysis storage engine includes: data rows divided into one or more segments, each segment having data columns corresponding to a data row; and bitmap data embedded in one or more segments, wherein the bitmap data is associated with the data columns and configured to locate target data in a query within one or more segments.

[0005] Furthermore, embodiments of this disclosure also provide a non-transitory computer-readable medium storing a set of instructions executable by one or more processors of a data analytics storage engine to cause the data analytics storage engine to initiate a method comprising: receiving a query to locate target data in the data analytics storage engine, the data analytics storage engine comprising: data rows divided into one or more segments, the one or more segments having data columns corresponding to the data rows; and bitmap data embedded in one or more segments, wherein the bitmap data is associated with the data columns; and locating the target data using the bitmap data in the one or more segments. Attached Figure Description

[0006] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the detailed description, explain the principles of the invention.

[0007] Figure 1A An exemplary format for storing row data in row-oriented storage is shown.

[0008] Figure 1B An exemplary format for storing column data in columnar storage is shown.

[0009] Figure 1C An exemplary format for storing consecutive columns of data in columnar storage is shown.

[0010] Figure 2 An exemplary format for columnar storage partitioning is shown.

[0011] Figure 3 An exemplary scan operation is shown in the data analytics storage engine.

[0012] Figure 4 A schematic diagram of an exemplary server for a data analytics storage engine according to some embodiments of the present disclosure is shown.

[0013] Figure 5 An exemplary column data block according to some embodiments of this disclosure is shown.

[0014] Figure 6 An exemplary segmentation with embedded bitmap data is shown according to some embodiments of the present disclosure.

[0015] Figure 7 An exemplary segmentation with embedded bitmap data and embedded dictionary data is shown according to some embodiments of the present disclosure.

[0016] Figure 8 An exemplary process for performing a query using embedded dictionary data and bitmap data according to some embodiments of this disclosure is shown.

[0017] Figure 9 A flowchart is shown of an exemplary method for performing a query using embedded bitmap data in a segment, according to some embodiments of the present disclosure.

[0018] Figure 10 A flowchart illustrating an exemplary method for performing a query using embedded bitmap data and dictionary data according to some embodiments of this disclosure is shown.

[0019] Figure 11 A flowchart illustrating an exemplary method for performing a query using embedded bitmap data and dictionary data according to some embodiments of this disclosure is shown.

[0020] Figure 12Exemplary procedures for performing queries using embedded dictionary data, bitmap data, and bitmap index data, according to some implementations of this disclosure, are shown.

[0021] Figure 13 A flowchart illustrating an exemplary method for performing a query using embedded bitmap data, dictionary data, and bitmap indexes according to some embodiments of this disclosure is shown. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. Unless otherwise stated, the following description refers to the accompanying drawings, wherein the same reference numerals in different drawings denote the same or similar elements. The embodiments set forth in the following description of exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present invention as set forth in the appended claims.

[0023] Many modern data analytics storage engines or databases store data in a column-based rather than row-based manner. Figure 1A An exemplary format for storing row data using row-based storage is shown. For example... Figure 1A As shown, the data can be logically represented as a two-dimensional table, including rows and columns. Figure 1A The table shown has 4 columns and 6 rows. These columns are named "Employee ID," "Last Name," "First Name," and "Salary." The data displayed in the table can be stored in several ways. One method is called row-based storage. In row-based storage, data is stored row by row, with all columns of a single row physically placed together, similar to... Figure 1A The data shown is row-oriented storage. Row-oriented storage is used to efficiently access data located in the same row. For example, if a user of a data analytics storage engine system wants to retrieve all column information associated with the entry "100" in the "Employee ID" column, the user can easily retrieve the entries "Smith", "Joe", and "20000" from storage because these entries are physically stored together. Row-oriented storage is often used for transactional queries, such as Online Transactional Processing (OLTP).

[0024] Another way to store data is called columnar storage. In columnar storage, data is stored column by column, with all rows of a single column physically placed together. Figure 1B An exemplary format for storing columnar data in columnar storage is shown. For example... Figure 1BAs shown, each column is stored separately. For example, all entries in the "Employee ID" column are stored together. Sometimes, each column is further divided into blocks, and each block is stored in a compressed form. During query processing, data is read from storage back into memory in blocks.

[0025] In some implementations, columns can be stored together in sequence. Figure 1C An exemplary format for storing consecutive columns of data in columnar storage is shown. For example... Figure 1C As shown, each column is stored sequentially, one after another. For example, with... Figure 1A The same order presented in the text Figure 1C The columns are stored in the order of "Employee ID", "Last Name", "First Name", and "Salary".

[0026] Columnar storage is used to efficiently support analytical queries that are typically interested in a subset of one or more columns. Using columnar storage, data from specific columns or columns can be retrieved without wasting input / output bandwidth on unnecessary columns. Furthermore, columnar storage enables more efficient data compression because the data in a column is often of the same type. Columnar storage has proven to significantly reduce I / O costs for many analytical queries, particularly for Online Analytical Processing (OLAP).

[0027] To achieve better parallelism for columnar storage scans, some columnar stores employ file partitioning or a hybrid row-column storage approach. First, rows are divided into file partitions, which can be further divided into row groups. A file partition can comprise a complete set of rows for each column. Columnar storage is then used for each partition. Figure 2 This illustrates an exemplary format for partitioning in columnar storage. Figures 1A to 1C Taking the data shown in the figure as an example, Figure 2 The row-group columnar storage in the database divides the data into two partitions named Partition 1 and Partition 2. Partition 1 contains the first three rows of data, and Partition 2 contains the next three rows. The data in each row group is then stored according to the columnar storage method.

[0028] Split columnar storage gains many benefits from columnar storage applied to each file split, because the data within each split is still stored in columnar storage. In the following description, columnar storage is used to describe pure columnar storage and its row-column variants, and split columnar storage and hybrid row-column storage are used interchangeably.

[0029] An index is a data structure technique that can accelerate query processing in analytical storage engines. An index maps values ​​from one or more columns in an analytical storage engine table to the "Employee ID" value in the corresponding row. Indexes allow for quick lookups of rows with values ​​in a given column. Many major analytical storage engines support certain types of indexes. For example, many popular analytical storage engines widely support B-tree indexes and bitmap indexes.

[0030] To support indexing of data analytics storage engine tables, each row in a data analytics storage engine table can have a unique row number as its identifier. One of the most logical methods for assigning row numbers is to start numbering each row from the beginning of the file split and move downwards.

[0031] The scan operation is a primitive operation in SQL queries. A scan operation takes a table, an optional set of projection columns, and a set of predicates as input, and outputs a set of projected rows from the table that satisfy the given predicates. Predicates can be conditional expressions that evaluate to Boolean values. For example, in SQL queries, predicates can be encountered in the "where" clause and can be used to filter data. Figure 3 An exemplary scan operation in the data analytics storage engine is shown. According to Figure 3 This scan has the predicate "salary > 40000" and a projection (employee ID, salary). To perform the scan, each row of the table needs to be examined to check if it satisfies the predicate. If the row does, the selected column in the projection is output. For example, the second row of the "salary" column shows 50000, satisfying the predicate "salary > 40000". Therefore, the output is (101, 50000). The data in the selected column can also be referred to as the target data of the query.

[0032] Sometimes, certain predicates can be "push down" to the location where the data is stored. This optimization can significantly reduce query processing time by filtering data earlier rather than later. According to this processing framework, predicate pushdown can optimize queries by filtering data before it is transmitted over the network or loaded into memory. Predicate pushdown can also reduce processing time by skipping the reading of entire data files or blocks.

[0033] A min-max index can be part of an index, providing statistics on the range of values ​​for a column. Min-max indexes can be used for columns within data blocks at the file, partition, or row group levels. For a given predicate, a min-max index can be used to skip portions of the data file because it allows the data analytics storage engine to evaluate whether the predicate queries values ​​within the min-max range for a specific column.

[0034] A Bloom filter is a space-efficient probabilistic data structure used to test whether an element is a member of a set. For a given set, a Bloom filter indicates whether a value is absolutely not in the set, or possibly is. Therefore, false positive matches are possible in a Bloom filter, but false negative matches are not. Bloom filters facilitate predicate judgment by quickly indicating with relatively high accuracy whether a given key can be found in a column. Thus, data systems can quickly determine whether a portion of a column contains a given key and skip most columns that are determined not to contain the given key.

[0035] like Figure 2 As shown, columnar storage formats are already used in the big data ecosystem. One of the advantages of columnar storage formats is the efficient encoding and compression of columns. Furthermore, columnar storage formats are advantageous for sequential scans during query processing. For example, min-max indexes and Bloom filters can help indicate whether certain values ​​are stored in data blocks without spending significant time and resources reading those blocks. However, min-max indexes may not always be effective. For instance, if a column is not sorted, the min-max indexes for different parts of that column may overlap, forcing the data analytics storage engine to read more irrelevant data blocks that are later excluded because they do not satisfy the filter predicate. Additionally, Bloom filters are designed for queries with equivalent predicates, but their effectiveness largely depends on the false positive rate associated with probabilistic data structures. Therefore, a high false positive rate can negatively impact scan skipping effectiveness.

[0036] To compensate for these shortcomings, many traditional data analytics storage engine systems utilize bitmap indexes as secondary indexes. Bitmap indexes answer queries by performing bitwise logical operations on the bit data stored in a bitmap of a column. Bitmap indexes are traditionally considered suitable for low-cardinality columns, which have a moderate number of distinct values. An extreme case of low cardinality is Boolean data, which has only two distinct values ​​(e.g., true or false). Bitmap indexes are very effective at improving the query performance of data analytics storage engines. For example, bitmap indexes can provide fast access to equality predicate pattern matches (e.g., predicates that include equivalent literal values).

[0037] In addition to bitmap indexes, dictionary data can also be used. Dictionary data can include distinct values ​​for a corresponding column and inherit the data type of that column. For example, if a column contains string values, the corresponding dictionary data can be constructed using the string data type. Furthermore, a data column may contain multiple entries with the value "Smith". Therefore, the corresponding dictionary data can include only one "Smith" value, because dictionary data can include only distinct values ​​in the corresponding column.

[0038] The current design of bitmap indexes has several issues. For columns with relatively high cardinality, significantly more resources (e.g., storage resources) are required to maintain them. This is because bitmap indexes cannot be effectively compressed. Therefore, they require a large amount of memory to store. When using bitmap indexes, the data analytics storage engine system needs to load them from physical storage into memory, resulting in significant I / O overhead. If the data analytics storage engine system performs frequent data updates (e.g., data insertion and deletion), the associated bitmap indexes will also be loaded frequently, adding further pressure to the system. Therefore, bitmap indexes are only suitable for read-only tables or tables that are not frequently updated.

[0039] To address these issues, embodiments of this disclosure provide support for embedded bitmap indexes in columnar storage formats. Figure 4 A schematic diagram of a data analysis engine according to some embodiments of the present disclosure is shown. Figure 4 Cloud 180 includes one or more servers, including server 110 of data analytics engine 100. Server 110 includes bus 112 or other communication mechanisms for transmitting information, and one or more processors 116 communicatively coupled to bus 112 to process information. Processor 116 may be, for example, one or more microprocessors. In some embodiments, Docker may be used to deploy the data analytics storage engine.

[0040] Server 110 can transmit data to or communicate with another server 130 via network 122. Network 122 can be a local network, an internet service provider, the internet, or any combination thereof. Server 110's communication interface 118 is connected to network 122. Furthermore, server 110 can be coupled to peripheral devices 140 via bus 112. Peripheral devices 140 include displays (e.g., cathode ray tube (CRT), liquid crystal displays (LCD), touchscreens, etc.) and input devices (e.g., keyboards, mice, soft keyboards, etc.).

[0041] Server 110 can be made into a dedicated machine by using custom hardwired logic, one or more ASICs or FPGAs, firmware, or program logic combined with the server.

[0042] Server 110 also includes storage device 114, which may include memory 161 and physical storage 164 (e.g., hard disk drive, solid-state drive, etc.). Memory 161 may include random access memory (RAM) 162 and read-only memory (ROM) 163. Storage device 114 may be communicatively coupled to processor 116 via bus 112. Storage device 114 may include main memory, which may be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 116. After these instructions are stored in a non-transitory storage medium accessible to processor 116, server 110 becomes a dedicated machine customized to perform the operations specified in the instructions. As used herein, the term "non-transitory medium" means any non-transitory medium that stores data or instructions that cause a machine to operate in a particular manner. Such non-transitory medium may include non-volatile or volatile media. Non-transitory media include, for example, optical discs or disks, dynamic memory, floppy disks, foldable disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROM, EPROM, FLASH-EPROM, NVRAM, flash memory, registers, caches, any other memory chips or cassettes, and networked versions of the above components.

[0043] Various forms of media may be involved when transmitting one or more sequences of instructions to processor 116 for execution. For example, the instructions may initially be executed on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to server 110 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and suitable circuitry may place the data on bus 112. Bus 112 transmits the data to main memory within storage device 114, from which processor 116 retrieves and executes the instructions.

[0044] In the data analytics storage engine, data can be stored in data storage 170, which can be accessed by a server (e.g., server 110 or server 130) via network 122. In some embodiments, data can be stored in storage device 114 or physical storage 164 of server 110 or server 130. In some embodiments, data is stored in an entity separate from the server. For example, such as... Figure 4 As shown, the data can be stored in a data storage device 170 that is separate from the server (e.g., server 110 or server 130).

[0045] In some embodiments, bitmap indexes can be stored along with the associated data in a data file. Embedded bitmap index designs offer several advantages. First, the maintenance cost of storing and using bitmap indexes is very low. This is because data files are immutable. For example, columnar data files are typically stored in cloud storage or distributed file systems, which gives columnar data files their immutable properties. Therefore, when data is updated (e.g., data insertion or data deletion), no special processing is required to maintain the bitmap index. A separate delete index can be created to handle data updates. Second, when columnar or row-group columnar data and their corresponding bitmap indexes are stored close to each other, a mapping between the columnar or row-group columnar data and the corresponding bitmap indexes can be easily established. This eliminates the need for additional index metadata and file management data. Third, partitioned files can be read in parallel. Therefore, bitmap indexes embedded in partitioned files can also be read or analyzed in parallel, making the system more efficient when executing queries.

[0046] Typically, columnar storage formats in the big data ecosystem include one or more partitions. A partition can be a set of rows representing all columns. In some embodiments, a partition can be used as a unit for parallel reading. A partition can include column block data and column index data. Column block data can include column blocks for each column, and these column blocks can be contiguous. For example, a column block can be a contiguous compressed block of data for a single column, and for all columns, one column block can be followed by another, because the data is stored column-wise. Figure 2 Taking the data analysis storage engine shown as an example, row group 1 can be column block data, which includes 4 column blocks, corresponding to the "Employee ID", "Last Name", "First Name", and "Salary" columns respectively. In addition to the splits, columnar files usually include split footnotes and file footnotes at the end of the columnar file.

[0047] Figure 5 Exemplary column data blocks according to some embodiments of this disclosure are shown. Figure 5 As shown, partition 500 may include column block data 510. Column block data 510 may include one or more compressed column data blocks, such as column data 511, 512, 513, and 514. In some embodiments, column data 511, 512, 513, and 514 may be stored sequentially. For example, refer to [reference]. Figures 1A to 1C The "Employee ID", "Last Name", "First Name", and "Salary" columns can be stored sequentially. Therefore, the "Employee ID" column can be... Figure 5 In column 511, the "surname" column can be... Figure 5 The data in column 512, the "Name" column can be... Figure 5 In the data column 513, the "salary" column can be... Figure 5 The column data is 514. For example... Figure 5 As shown, column data 511, 512, 513, and 514 are stored sequentially, just like... Figure 1C The "Employee ID", "Last Name", "First Name", and "Salary" columns shown are the same.

[0048] The 500 partition can also include column index data 610 (in Figure 5 (Displayed in gray). In some embodiments, column data for each column is first written to storage, and then column index data for each corresponding column is written to storage. Column index data 610 may include index data of the column data stored in column block data 510. For example, column index data 610 may include index data of column data 511, 512, 513, 514, etc. In some embodiments, column data (e.g., column data 511, 512, 513, and 514) may include one or more row groups, and index data for each row group (e.g., the position of the row group, the minimum-maximum index of the row group, etc.) may be stored in column index data 610. In some embodiments, the index data stored in column index data 610 has the same order as the column data in column block data 510. For example, as Figure 5 As shown, column index data 610 may include sequentially stored index data 611, 612, 613, and 614. Index data 611 may correspond to column data 511. Index data 612 may correspond to column data 512. Index data 613 may correspond to column data 513. And index data 614 may correspond to column data 514.

[0049] In some embodiments, column index data may include one or more types of indexes for logical row groups (e.g., every 8000 rows). Column index data may include min-max statistics or Bloom filter entries for each row group. They can form segmented min-max indexes and Bloom filter indexes. For example, column index data 610 may include a min-max index of column block data 510. More specifically, index data 614 may include a min-max index of the columns stored in column data 514. Figures 1A to 1C For example, column data 514 can include the "salary" column, and index data 614 can include the minimum and maximum indexes of the "salary" column, with a minimum value of "20000" and a maximum value of "90000".

[0050] In some embodiments, segmentation may further include segmentation footnotes. For example, Figure 5 The split 500 in the document may also include a split footnote 710. A file footnote 720 may be included at the end of the document following the last file split in the list document.

[0051] In some embodiments, the embedded bitmap index can be stored as an embedded table. Figure 6Exemplary segmentation with embedded bitmap data is shown according to some embodiments of this disclosure. Similar to Figure 5 The division shown is 500. Figure 6 The partition 600 shown may also include column block data 510, column index data 610, partition footnotes 710, and file footnotes 720. Furthermore, partition 600 includes bitmap column block data 530. In some embodiments, bitmap column block data 530 corresponds to column block data 510. For example, bitmap column block data 530 includes bitmap indexes of column data (e.g., column data 511, 512, 513, 514, etc.) stored in column block data 510. In some embodiments, there is a one-to-one mapping between columns within the partition and their corresponding bitmap index data. For example, column col1 may have corresponding embedded bitmap index data, and column col2 may also have corresponding embedded bitmap index data, with the embedded bitmap index data of column col2 being separate from that of column col1. In some embodiments, bitmap column block data 530 may be stored close to column block data 510. For example, bitmap column block data 530 may be stored immediately before or immediately after column block data 510. In some embodiments, a column of data can be stored close to the embedded bitmap index data corresponding to that column. Storing column data together with its corresponding bitmap index data can provide more efficient data processing because the system can traverse between the data column and the bitmap index data more efficiently.

[0052] In some embodiments, segment 600 may include bitmap column index data 630. Bitmap column index data 630 may correspond to bitmap column block data 530. For example, bitmap column index data 630 may include a minimum-maximum index of bitmap column block data 530. In some embodiments, bitmap column index data 630 may be stored close to column index data 610. For example, bitmap column index data 630 may be stored immediately before or immediately after column index data 610.

[0053] In some embodiments, the embedded table may include two types of indexes, such as dictionary data and bitmap data. Figure 7 An exemplary segmentation with embedded bitmap data and embedded dictionary data is shown according to some embodiments of this disclosure. Similar to Figure 5 The division shown is 500 and Figure 6 The division shown is 600. Figure 7 The partition 700 shown can also include column block data 510, column index data 610, partition footnote 710, or file footnote 720. Similar to... Figure 6The partitions 600 and 700 shown may further include bitmap column block data 530 and bitmap column index data 630. Additionally, partition 700 may include dictionary column block data 520. In some embodiments, a dictionary column (e.g., column data 521) stored in dictionary column block data 520 may include all distinct values ​​of the corresponding column data. In some embodiments, the dictionary column may inherit the data type of the corresponding column data and retain the order of the corresponding column data. In some embodiments, a bitmap column (e.g., column data 531) stored in bitmap column block data 530 may include bitmaps of one or more values ​​of the corresponding dictionary column (e.g., a dictionary column in the same embedded table). In some embodiments, the bitmap column may be stored as a binary column. In some embodiments, dictionary column block data 520 may be searchable (e.g., linear search or binary search). It is understood that column block data 510, dictionary column block data 520, bitmap column block data 530, column index data 610, and bitmap column index data 630 may be stored in any order.

[0054] In some embodiments, bitmap indexes (e.g., Figure 6 and Figure 7 The bitmap column block data (530) shown can be a roar bitmap. A roar bitmap can be one where the data is divided into integer blocks (e.g., [0, 2...). 16 ),[2 16 2x 2 16 Roaring bitmaps are compressed bitmaps (e.g., uncompressed bitmaps, simple lists of integers, or run lists). Within a block, uncompressed bitmaps, simple lists of integers, or run lists can be used. Regardless of the format used, roaring bitmaps allow for fast checks of the existence of values ​​(e.g., using binary search). As a result, roaring bitmaps can compute many operations faster than run-length encoded formats (e.g., Word Aligned Hybrid (WAH), Enhanced World Aligned Hybrid (EWAH), Concise, etc.). Furthermore, roaring bitmaps can offer better compression ratios.

[0055] like Figure 7 As shown, bitmap indexes can be embedded in a column data file or a row group column file along with the column data itself. In some embodiments, columns can also have a dictionary containing all distinct values. With the help of the dictionary, the data analytics storage engine system can load only the bitmap corresponding to a specific key, so that only the portion of the data file containing matching records is scanned. Another advantage of using embedded bitmap indexes is the ability to answer queries by performing bitwise logical operations on the bitmap. Compared to other index structures, it exhibits higher efficiency in terms of space and performance.

[0056] Figure 8Exemplary processes for performing queries using embedded dictionary data and bitmap data according to some embodiments of this disclosure are shown. Figure 8 As shown, the exemplary query to be executed is "select sum(col3) from table1 where col1 = 'R'". That is, this query selects the rows in column col1 where the value is "R" and obtains the sum of all values ​​in the selected rows in column col3. It can be understood that any query designed to find specific target data in a data analytics storage engine can use this query. Figure 8 The procedure shown is used to execute the query. For example, the target data in the query "select sum(col3) from table1 where col1 = 'R'" is the data in column 1 of column col3 where the value is "R". This is understandable. Figure 8 The process shown can be performed by a data analytics storage engine system (e.g., Figure 4 The data analysis storage engine system 100 or server (e.g., Figure 4 The server 110 in the middle is executed.

[0057] To select rows where "col1 = 'R'", the data analysis storage engine system executing the query can load... Figure 7 The dictionary column data 520 is shown. When a query is executed, the data analysis storage engine system can scan the dictionary column data 520 (e.g., sequential scan or binary search) and locate the predicate to find its corresponding encoded value (e.g., col1 = "R"). In some embodiments, the dictionary column data 520 is scanned to find mapping information between predicates and their corresponding encoded values.

[0058] In some embodiments, because the dictionary columns contain distinct values, there is only one "R" value in the dictionary data block. In some embodiments, the dictionary columns are sorted and maintained in order. Therefore, a binary search can be performed on the dictionary columns to find entries that contain distinct "R" values.

[0059] like Figure 8As shown, blocks in dictionary data block 820 of col1 can include position information of corresponding blocks in bitmap data block 830 of col1. That is, blocks in dictionary data block 820 of col1 can include a mapping between predicate values ​​(e.g., "R" in column col1) and corresponding blocks in bitmap data block 830 of col1. For example, a third entry in dictionary data block 820 of col1 can include a pointer to a corresponding block in bitmap data block 830 of col1 (e.g., the third block in bitmap data block 830 of col1). In some embodiments, the mapping between predicate values ​​and corresponding blocks, as well as offsets within blocks in the bitmap data, can be established based on encoded values ​​of different values ​​in the dictionary column (e.g., the value "R" in column col1). In some embodiments, entries in blocks of dictionary data block 820 of col1 can also include offsets. For example, the value "R" in the third block can include an offset (e.g., the value 3) indicating the relative position of the value "R" within the block. This offset can be used to locate entries (e.g., the third entry in the corresponding bitmap data block) within the corresponding bitmap data block. Figure 8 In the example shown, the entry containing the value “R” has an offset pointing to an entry in the bitmap data block containing values ​​such as “1”, “3”, “20001”, “20006”. In some embodiments, instead of an offset, the value in the block of dictionary data block 820 of col1 may include a pointer to the corresponding entry in the corresponding bitmap data block.

[0060] As described above, a mapping relationship can exist between the dictionary data block 820 and the bitmap data block 830 of col1. This mapping relationship allows the data analysis storage engine system to quickly locate relevant entries in the bitmap data block 830 of col1 that satisfy the predicate (e.g., col1 = "R"). Therefore, the data analysis storage engine system does not need to load and access irrelevant blocks in the bitmap data block 830 of col1. Furthermore, the data analysis storage engine system does not need to load and access irrelevant entries in the corresponding bitmap data blocks (e.g., the third block in the bitmap data block 830 of col1). Therefore, the data analysis storage engine system can speed up query execution and conserve valuable I / O resources during query execution.

[0061] like Figure 8As shown, when an entry in the bitmap data block 830 of col1 is determined based on a predicate (e.g., col1 = "R"), the entry can be loaded into memory and accessed. Bitwise operations can be performed on the values ​​stored in the entry to determine the location of the data block containing the relevant data to be queried (e.g., data stored in column col3). For example, a bitwise operation can be performed on the value "3" stored in the entry to determine the location of data block 841. Data block 841 may include data from column col3, where the corresponding rows satisfy the predicate (e.g., col1 = "R"). Similarly, a bitwise operation can be performed on the value "20001" in the entry to determine the location of data block 842. Therefore, the data analytics storage engine system can access only the relevant data blocks in the partition. The data analytics storage engine system can ignore irrelevant blocks in data block 840 of col3. Furthermore, in some embodiments, the data analytics storage engine system can ignore all blocks in data block 850 of col2 and data block 860 of col1.

[0062] In some embodiments, Figure 8 The bitmap in the image can be a roaring bitmap. A roaring bitmap divides bit entries into blocks of integers (e.g., [0, 2...). 16 ),[2 16 2x 2 16 Within a block, uncompressed bitmaps, simple lists of integers, or run lists can be used. Regardless of the format used, roar bitmaps allow for fast checks of the existence of any value (e.g., using binary search). As a result, roar bitmaps can compute many operations faster than run-length encoded formats (e.g., Word Aligned Hybrid (WAH), Enhanced World Aligned Hybrid (EWAH), Concise, etc.). Furthermore, roar bitmaps can offer better compression ratios.

[0063] This disclosure provides a method for performing queries using embedded bitmap data in a segmented process. Figure 9 A flowchart illustrating an exemplary method for performing queries using embedded bitmap data in segmentation according to some embodiments of this disclosure is shown. It will be understood that... Figure 9 Method 9000 in the middle can be provided by a data analysis storage engine system (e.g., Figure 4 The data analysis storage engine system 100 or server (e.g., Figure 4 The server (110) in the middle executes. It is also understandable that... Figure 9 Method 9000 can be used with embedded bitmap data (e.g., Figure 6 and Figure 7Bitmap column block data (530) or bitmap index data (e.g., Figure 6 and Figure 7 The data analysis storage engine runs on the bitmap column index data (630).

[0064] In step 9010, a query is received to locate target data in the column data. In some embodiments, the query includes a predicate that sets conditions on the target data. Figure 8 Taking the process as an example, the query "select sum(col3) from table1 where col1='R'" has the predicate "col1='R'", and the target data is the data in column col3 where the value of column col1 is equal to "R". In some embodiments, the query can be generated by the user of the data analysis storage engine system or by the data analysis storage engine system itself.

[0065] In some embodiments, step 9020 may be performed after step 9010. In step 9020, bitwise operations are performed on one or more values ​​stored in the bitmap data to locate the target data. For example, such as Figure 8 As shown, the value "3" stored in the entry can be bitwise operated on to determine the location of data block 841. In some embodiments, the bitmap data includes a roar bitmap.

[0066] In step 9030, bitmap data embedded in the segment is used to locate the target data. In some embodiments, the bitmap data is associated with column data in the segment. In some embodiments, locating the target data is performed by locating a data block containing the target data using the bitmap data. The target block is then accessed (e.g., loaded into memory) to locate the target data.

[0067] This disclosure also provides a method for performing queries using embedded bitmap data and dictionary data. Figure 10 A flowchart illustrating an exemplary method for performing a query using embedded bitmap data and dictionary data according to some embodiments of this disclosure is shown. In addition... Figure 9 In addition to steps 9010, 9020 and 9030 shown, Figure 10 Method 9100 further includes step 9013. It is understood that... Figure 10 Method 9100 in the middle can be used by a data analysis storage engine system (e.g., Figure 4 The data analysis storage engine system 100 or server (e.g., Figure 4 The server (110) in the middle executes. It is also understandable that... Figure 10 Method 9100 can be used with embedded bitmap data (e.g., Figure 6 and Figure 7 Bitmap column block data 530 or Figure 8 The bitmap data block 830 of col1 in the dictionary (e.g., ...) or dictionary data (e.g., ...) Figure 7 The dictionary column block data 520 or Figure 8 It runs on the data analysis storage engine of the dictionary data block 820 of col1.

[0068] In step 9013, one or more values ​​stored in the bitmap data are located. These one or more values ​​correspond to the positions of the target data in the segmentation. In some embodiments, the dictionary data includes mapping information for predicate values ​​and one or more values ​​in the query. For example, such as... Figure 8 As shown, the blocks in dictionary data block 820 of col1 may include a mapping between predicate values ​​(e.g., "R" in the col1 column) and corresponding blocks in bitmap data block 830 of col1. More specifically, the third entry in dictionary data block 820 of col1 may include a pointer to the corresponding block in bitmap data block 830 of col1 (e.g., the third block in bitmap data block 830 of col1).

[0069] This disclosure also provides a method for performing queries using embedded bitmap data and dictionary data. Figure 11 A flowchart illustrating an exemplary method for performing a query using embedded bitmap data and dictionary data according to some embodiments of this disclosure is shown. In addition... Figure 9 or Figure 10 Apart from steps 9010, 9020, 9030, and 9013 shown, Figure 11 Method 9200 further includes step 9011. It is understood that... Figure 11 Method 9200 in the middle can be used by a data analysis storage engine system (e.g., Figure 4 Data analysis storage engine system 100) or server (e.g., Figure 4 The server (110) executes this. It's also understandable that... Figure 11 Method 9200 can be used with embedded bitmap data (e.g., Figure 6 and Figure 7 Bitmap column block data 530), bitmap index data (e.g., Figure 6 and Figure 7 It runs on the bitmap column index data (630) or the bitmap data analysis storage engine.

[0070] In step 9011, the predicate value in the query is located in the dictionary data using dictionary data. In some embodiments, the dictionary data can be scanned (e.g., sequential scan or binary search) to locate the predicate value and its corresponding encoded value, which can be directly mapped to the bitmap entry in the specific location tile and the offset in the tile.

[0071] In some embodiments, an index of bitmap data can be used to perform a query. Figure 12 Exemplary procedures for performing queries using embedded dictionary data, bitmap data, and bitmap index data, according to some embodiments of this disclosure, are illustrated. For example... Figure 12 As shown, in Figure 8 Above the process shown, there can also be bitmap index block 825 for col1. It is understandable that any query aimed at finding specific target data in the data analytics storage engine can use this. Figure 12 The procedure shown is used to execute the query. For example, the target data in the query "select sum(col3) from table1 where col1='R'" is the data in column col3 where the value in column col1 is equal to "R". This is understandable. Figure 12 The process shown can be performed by a data analytics storage engine system (e.g., Figure 4 The data analysis storage engine system 100 or server (e.g., Figure 4 The server 110 in the middle is executed.

[0072] like Figure 12 As shown, when a predicate is found in dictionary data block 820 of col1 (e.g., col1 = "R"), the system can search in bitmap index block 825 of col1 to find the predicate's location information in the corresponding bitmap data block. For example, as Figure 12 As shown, the system can search bitmap index block 825 of col1 and find the offset corresponding to the predicate. Using the offset, the system can easily find the entry in the corresponding bitmap data block (e.g., the third entry in the corresponding bitmap data block). Figure 12 In the example shown, the entry containing the value “R” has an offset pointing to an entry in the bitmap data block that includes values ​​such as “1”, “3”, “20001”, and “20006”. In some embodiments, instead of the offset, the location information stored in the bitmap index block 825 of col1 may include a pointer to the corresponding entry in the corresponding bitmap data block.

[0073] This disclosure also provides a method for performing queries using embedded bitmap data, dictionary data, and bitmap indexes. Figure 13 A flowchart illustrating an exemplary method for performing a query using embedded bitmap data, dictionary data, and a bitmap index, according to some embodiments of this disclosure, is shown. In addition... Figure 11 In addition to steps 9010, 9020, 9030, and 9011 shown, Figure 13 Method 9300 further includes steps 9012 and 9014. It is understood that... Figure 13 Method 9300 in the middle can be used by a data analysis storage engine system (e.g., Figure 4 The data analysis storage engine system 100 or server (e.g., Figure 4 The server (110) in the middle executes. It is also understandable that... Figure 13 Method 9300 can be used with embedded bitmap data (e.g., Figure 6 and Figure 7 Bitmap column block data 530), bitmap index data (e.g., Figure 6 and Figure 7 It runs on the bitmap column index data (630) or the bitmap data analysis storage engine.

[0074] In step 9012, the embedded bitmap index data is searched to locate the position information of the predicate values ​​in the embedded bitmap data. For example, such as Figure 12 As shown, the system can search bitmap index block 825 of col1 and find the offset corresponding to the predicate. Using the offset, the system can easily locate the entry in the corresponding bitmap data block (e.g., the third entry in the corresponding bitmap data block). In some embodiments, instead of the offset, the location information stored in the bitmap index data may include a pointer to the corresponding entry in the corresponding bitmap data.

[0075] In step 9014, one or more values ​​stored in the bitmap data are located based on the location information. These values ​​correspond to the positions of the target data during segmentation. For example, such as... Figure 11 As shown, the location information found in the bitmap index data can be used to locate the corresponding entry in the bitmap data block 830 of col1.

[0076] Some embodiments of this disclosure offer numerous advantages over conventional designs of data analytics storage engines or databases. For example, bitmap indexes introduced in some embodiments can use bitmaps and answer queries by performing bitmap logic operations on these bitmaps. Therefore, the data analytics storage engine can reduce space consumption and logic operation overhead. Roaring bitmaps are also more efficient, thus resolving the previous limitations on high-cardinality columns.

[0077] Furthermore, in data analytics storage engines within cloud environments, columnar data files are typically stored across one or more cloud storage systems or one or more distributed file systems. In some embodiments, columnar data files have immutable properties, which also applies to embedded bitmap indexes. Therefore, there is no additional overhead specifically associated with maintaining bitmap indexes when performing update / delete operations. The limitations of traditional secondary bitmap indexes on frequently updated tables are no longer an issue.

[0078] Furthermore, bitmap indexes introduced in some embodiments go beyond simple min-max indexes where validity depends on the ordered arrangement of the corresponding columns. Bitmap indexes also differ from simple Bloom filters, which can return many false positives to queries. In some embodiments, bitmap indexes can provide the exact location of key storage. Therefore, embedded bitmap indexes can be used to compensate for min-max indexes to achieve efficient scans and provide significant efficiency in terms of storage and performance.

[0079] It is understood that the above embodiments can be implemented by hardware, software (program code), or a combination of hardware and software. If implemented by software, the software can be stored in the aforementioned computer-readable medium. When executed by a processor, the software can perform the disclosed methods. The computing units and other functional units described in this disclosure can be implemented by hardware, software, or a combination of hardware and software. It is understood that multiple modules / units described above can be merged into one module / unit, and each of the above modules / units can be further divided into multiple sub-modules / sub-units.

[0080] The embodiments may be further described using the following terms:

[0081] 1. A method for executing a query in a data analysis storage engine, the method comprising:

[0082] Receive queries to locate target data in the data analysis storage engine, the data analysis storage engine comprising:

[0083] Data rows divided into one or more segments, each segment having data columns corresponding to the data rows, and

[0084] Bitmap data embedded in one or more segments, wherein the bitmap data is associated with the data column; and

[0085] The target data is located using the bitmap data in one or more of the segments.

[0086] 2. The method according to Clause 1, wherein locating the target data using bitmap data from the one or more segments further comprises:

[0087] Perform bitwise operations on one or more values ​​stored in the bitmap data to locate the target data.

[0088] 3. The method according to Clause 2, wherein:

[0089] The data analytics storage engine also includes bitmap index data embedded in the one or more partitions, wherein the bitmap index data is associated with the bitmap data embedded in the one or more partitions; and

[0090] Locating the target data using the bitmap data in one or more segments further includes:

[0091] Use the bitmap index data to locate one or more values ​​stored in the bitmap data.

[0092] 4. The method according to any one of clauses 1-3, wherein:

[0093] The data columns in one or more segments are divided into data blocks; and

[0094] Locating the target data using the bitmap data in one or more segments further includes:

[0095] Use the bitmap data to locate the data block including the target data; and

[0096] Access the data block.

[0097] 5. The method according to any one of clauses 2-4, wherein:

[0098] The data analysis storage engine also includes dictionary data embedded in the one or more partitions, wherein the dictionary data is associated with the data columns; and

[0099] Locating the target data using the bitmap data in one or more segments further includes:

[0100] Use dictionary data to locate the one or more values ​​stored in the bitmap data.

[0101] 6. The method according to Clause 5, wherein:

[0102] The dictionary data includes mapping information between the predicate values ​​in the query and one or more of the values; and

[0103] Locating the one or more values ​​stored in the bitmap data using dictionary data also includes:

[0104] The mapping information is used to locate one or more values ​​stored in the bitmap data.

[0105] 7. The method according to any one of clauses 1-6, wherein:

[0106] The bitmap data is a roar bitmap.

[0107] 8. A data analysis storage engine system, comprising:

[0108] Data rows divided into one or more segments, each segment having data columns corresponding to the data rows, and

[0109] Bitmap data embedded in one or more segments, wherein the bitmap data is associated with the data column and the bitmap data is configured to locate target data in a query within one or more segments.

[0110] 9. The data analysis storage engine system as described in Clause 8, wherein the bitmap data is further configured as follows:

[0111] Perform bitwise operations on one or more values ​​stored in the bitmap data to locate the target data.

[0112] 10. The data analysis storage engine system as described in Clause 9, wherein:

[0113] The data analysis storage engine system also includes bitmap index data embedded in the one or more partitions, wherein the bitmap index data is associated with the bitmap data embedded in the one or more partitions; and

[0114] The bitmap index data is configured as follows:

[0115] Locate the one or more values ​​stored in the bitmap data.

[0116] 11. A data analysis storage engine system according to any one of Clauses 8-10, wherein:

[0117] The data columns in one or more segments are divided into data blocks; and

[0118] The bitmap data is also configured as follows:

[0119] Locate the data block containing the target data.

[0120] 12. A data analysis storage engine system according to any one of Clauses 9-11, wherein:

[0121] The data analysis storage engine system also includes dictionary data embedded in the one or more partitions, wherein the dictionary data is associated with the data columns; and

[0122] The dictionary data is configured as follows:

[0123] Use dictionary data to locate the one or more values ​​stored in the bitmap data.

[0124] 13. The data analytics storage engine system as described in Clause 12, wherein:

[0125] The dictionary data includes mapping information between the predicate values ​​in the query and one or more of the values; and

[0126] The dictionary data is also configured as follows:

[0127] The mapping information is used to locate one or more values ​​stored in the bitmap data.

[0128] 14. A data analysis storage engine system according to any one of Clauses 8-13, wherein:

[0129] The bitmap data is a roar bitmap.

[0130] 15. A non-transitory computer-readable medium storing an instruction set executable by one or more processors of a data analytics storage engine to cause the data analytics storage engine to initiate a method comprising:

[0131] Receive queries to locate target data in the data analysis storage engine, the data analysis storage engine comprising:

[0132] Data rows divided into one or more segments, each segment having data columns corresponding to the data rows, and

[0133] Bitmap data embedded in one or more segments, wherein the bitmap data is associated with the data column; and

[0134] The target data is located using the bitmap data in one or more of the segments.

[0135] 16. The non-transitory computer-readable medium according to Clause 15, wherein locating the target data using the bitmap data in the one or more segments further comprises:

[0136] Perform bitwise operations on one or more values ​​stored in the bitmap data to locate the target data.

[0137] 17. The non-transitory computer-readable medium as described in Clause 16, wherein:

[0138] The data analytics storage engine also includes bitmap index data embedded in the one or more partitions, wherein the bitmap index data is associated with the bitmap data embedded in the one or more partitions; and

[0139] The method further includes:

[0140] Use the bitmap index data to locate the one or more values ​​stored in the bitmap data.

[0141] 18. A non-transitory computer-readable medium according to any one of clauses 15-17, wherein:

[0142] The data columns in one or more segments are divided into data blocks; and

[0143] The method further includes:

[0144] Use the bitmap data to locate the data block including the target data; and

[0145] Access the data block.

[0146] 19. A non-transitory computer-readable medium according to any one of clauses 16-18, wherein:

[0147] The data analysis storage engine also includes dictionary data embedded in the one or more partitions, wherein the dictionary data is associated with the data columns; and

[0148] The method further includes:

[0149] Use dictionary data to locate the one or more values ​​stored in the bitmap data.

[0150] 20. The non-transitory computer-readable medium as described in Clause 19, wherein:

[0151] The dictionary data includes mapping information between the predicate values ​​in the query and the one or more values, and

[0152] The method further includes:

[0153] The mapping information is used to locate one or more values ​​stored in the bitmap data.

[0154] 21. A non-transitory computer-readable medium according to any one of clauses 15-20, wherein:

[0155] The bitmap data is a roar bitmap.

[0156] It should be noted that the relational terms such as "first" and "second" used in this document are only used to distinguish one entity or operation from another, and do not require or imply any actual relationship or order between these entities or operations. Furthermore, words such as "including," "having," "containing," and "comprising," as well as other similar forms, have equivalent meanings and are open-ended, meaning that the one or more items following one of these words are not intended to be an exhaustive list of these items, or are limited to only the listed items.

[0157] Unless otherwise expressly stated, the term "or" covers all possible combinations except where it is impractical. For example, if a component is declared to include A or B, then unless otherwise expressly stated or impractical, the component can include A, or B, or A and B. As a second example, if a component is declared to include A, B, or C, then unless otherwise expressly stated or impractical, the component can include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0158] In the foregoing description, embodiments have been described with reference to numerous specific details, which may vary depending on the implementation. Certain adjustments and modifications can be made to the described embodiments. Other embodiments will be apparent to those skilled in the art in light of the specification and practice of the invention disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the appended claims. The sequence of steps shown in the figures is also for illustrative purposes only and is not intended to limit to any particular order of steps. Therefore, those skilled in the art will understand that these steps may be performed in a different order when implementing the same method. Exemplary embodiments have been disclosed in the drawings and specification; however, many variations and modifications can be made to these embodiments. Therefore, although specific terminology is used, it is used only in a general descriptive sense and is not intended to be limiting; the scope of the embodiments is defined by the appended claims.

Claims

1. A method for executing a query in a data analysis storage engine, the method comprising: Receiving queries to locate target data in the data analysis storage engine, the data analysis storage engine supporting embedded bitmap indexes in a columnar storage format, the data analysis storage engine comprising: Data rows divided into one or more segments, each segment having data columns corresponding to the data rows, and Bitmap data and bitmap index data embedded in the one or more segments, wherein the bitmap data is associated with the data column, and the bitmap index data is associated with the bitmap data embedded in the one or more segments; and The target data is located using the bitmap data in one or more of the segments; Locating the target data using the bitmap data in one or more segments further includes: performing a bitwise operation on one or more values ​​stored in the bitmap data to locate the target data; Locating the target data using the bitmap data in one or more segments further includes: Use the bitmap index data to locate the one or more values ​​stored in the bitmap data.

2. The method according to claim 1, wherein: The data columns in one or more segments are divided into data blocks; and Locating the target data using the bitmap data in one or more segments further includes: Use the bitmap data to locate the data block containing the target data; as well as Access the data block.

3. The method according to claim 1, wherein: The data analysis storage engine also includes dictionary data embedded in one or more of the partitions, wherein the dictionary data is associated with the data columns; and Locating the target data using the bitmap data in one or more segments further includes: Use dictionary data to locate the one or more values ​​stored in the bitmap data.

4. The method according to claim 3, wherein: The dictionary data includes mapping information between the predicate values ​​in the query and one or more of the values; and Locating the one or more values ​​stored in the bitmap data using dictionary data also includes: The mapping information is used to locate one or more values ​​stored in the bitmap data.

5. The method according to claim 1, wherein: The bitmap data is a roar bitmap.

6. A data analysis storage engine system that supports embedded bitmap indexes in columnar storage format, the data analysis storage engine comprising: Data rows divided into one or more segments, each segment having data columns corresponding to the data rows, and Bitmap data and bitmap index data embedded in one or more segments, wherein the bitmap data is associated with the data column, the bitmap index data is associated with the bitmap data embedded in one or more segments, and the bitmap data is configured to perform bitwise operations on one or more values ​​stored in the bitmap data to locate target data; The bitmap index data is configured to locate the one or more values ​​stored in the bitmap data.

7. The data analysis storage engine system according to claim 6, wherein: The data columns in one or more segments are divided into data blocks; and The bitmap data is also configured as follows: Locate the data block containing the target data.

8. The data analysis storage engine system according to claim 6, wherein: The data analysis storage engine system also includes dictionary data embedded in the one or more partitions, wherein the dictionary data is associated with the data columns; and The dictionary data is configured as follows: Use dictionary data to locate the one or more values ​​stored in the bitmap data.

9. The data analysis storage engine system according to claim 8, wherein: The dictionary data includes mapping information between the predicate values ​​in the query and the one or more of those values; and The dictionary data is also configured as follows: The mapping information is used to locate one or more values ​​stored in the bitmap data.

10. The data analysis storage engine system according to claim 6, wherein: The bitmap data is a roar bitmap.

11. A non-transitory computer-readable medium storing an instruction set executable by one or more processors of a data analytics storage engine to cause the data analytics storage engine to initiate a method supporting embedded bitmap indexes in a columnar storage format, the method comprising: Receive queries to locate target data in the data analysis storage engine, the data analysis storage engine comprising: Data rows divided into one or more segments, each segment having data columns corresponding to the data rows, and Bitmap data and bitmap index data embedded in the one or more segments, wherein the bitmap data is associated with the data column, and the bitmap index data is associated with the bitmap data embedded in the one or more segments; and The target data is located using the bitmap data in one or more of the segments; Locating the target data using the bitmap data in one or more segments further includes: Perform bitwise operations on one or more values ​​stored in the bitmap data to locate the target data; The method further includes: Use the bitmap index data to locate the one or more values ​​stored in the bitmap data.

12. The non-transitory computer-readable medium according to claim 11, wherein: The data columns in one or more segments are divided into data blocks; and The method further includes: Use the bitmap data to locate the data block including the target data; and Access the data block.

13. The non-transitory computer-readable medium according to claim 11, wherein: The data analysis storage engine also includes dictionary data embedded in one or more of the partitions, wherein the dictionary data is associated with the data columns; and The method further includes: Use dictionary data to locate the one or more values ​​stored in the bitmap data.

14. The non-transitory computer-readable medium according to claim 13, wherein: The dictionary data includes mapping information between the predicate values ​​in the query and the one or more values, and The method further includes: The mapping information is used to locate one or more values ​​stored in the bitmap data.

15. The non-transitory computer-readable medium according to claim 11, wherein: The bitmap data is a roar bitmap.

Images