A columnar storage index method and device based on non-volatile memory

Abstract

The application discloses a columnar storage index method and device based on nonvolatile memory, relates to the field of computers, and aims to real-time update of an index column, establishes a cache pool in the NVM, utilizes the characteristics of the NVM, constructs Main and Delta structures through two-stage batch update, the Main comprises multiple Segments, each Segment contains metadata information and data information, the Delta comprises two Delta files, and the Delta files are alternately updated; a Bloom Filter is constructed for the Delta files in the memory to query and judge updated data. In view of the problem that the index column causes serious write amplification when being combined in the traditional segment, the application uses metadata information combination to replace the traditional columnar index combination when creating an index, so that the memory space occupation and read-write amplification are reduced under the condition that the read-write performance is not decreased. In view of the low-base dimension column, an inverted index is created for the column, so that the memory space occupation is greatly reduced and the equivalent query performance is improved.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically to a columnar storage indexing method and apparatus based on non-volatile memory. Background Technology

[0002] With the rapid development of the internet age and modern information technology, and the gradual maturation of social networks (Weibo, Facebook, Twitter), mobile networks, e-commerce, and various smart tools and services will generate more data than ever before in history. Data is experiencing explosive growth, and a new era—the era of big data—has arrived. However, the strategic significance of big data lies not only in mastering massive amounts of information. In this era where information is everything, how to quickly acquire and retrieve information has also become a key focus.

[0003] The organization and storage of data is a key focus of databases today. Traditional data organization and storage methods utilize row-based storage, such as SQL Server and Oracle. However, current queries are mainly based on specific fields for querying and outputting results. The query performance of traditional storage methods still cannot meet the requirements. Therefore, column-based storage has emerged. Column-based storage stores data belonging to the same field contiguously. Column-based storage requires only a few disk rotations to hit the data, reducing disk I / O overhead.

[0004] SAP HANA is an in-memory analytical database with columnar storage. Because it stores both data and indexes in memory, it offers high-performance data querying capabilities. Users can directly query and analyze large amounts of real-time business data. The columnar storage is compressed within the in-memory database, allowing more data to fit into memory. As an in-memory database, it efficiently provides data insertion, modification, and deletion functions. However, when the data volume is too large, memory capacity becomes limited, potentially leading to system crashes. Therefore, most storage engines currently employ a memory-disk hierarchical storage structure. Block devices, represented by disks or hard drives, have advantages such as non-volatility, large storage capacity, and relatively low cost. As the name suggests, block devices read and write in blocks and cannot perform byte addressing. The random and sequential read / write performance of block devices lags far behind that of memory, with the performance difference between random and sequential read / write operations in block devices being approximately tens of times. This makes random read / write performance a very expensive overhead in block devices.

[0005] Most columnar storage engines currently do not support absolute updates. Updates are mainly divided into out-of-place updates and in-place updates. In-place updates indirectly incorporate the new values ​​of the data into the old ones. For example, in MongoDB, in-place updates, especially for large documents, can reduce disk utilization by up to 7 times for some workloads. However, when faced with a large number of real-time updates, it leads to a large amount of random I / O and read / write amplification, greatly reducing read and write performance. Furthermore, multiple updates and deletions can severely cause disk page fragmentation, thus reducing space utilization. In-place updates also cause lock contention, affecting read and write efficiency. To avoid lock contention issues caused by in-place updates, Lucene's Doc Values ​​index provides the RCU (Read-Copy-Update) approach. First, the partition that needs to be updated is located; then, all data in that partition is copied; finally, the partition is rebuilt, and the updated values ​​are written to the new partition, replacing the old ones. However, this method can cause severe write amplification when facing real-time updates.

[0006] Off-site updates primarily involve merging old and new data and then importing them into columnar storage in batches. This makes real-time data analysis impossible, requiring waiting for hours or even days. For example, Hadoop Database doesn't locate rows to be deleted or updated; it only appends records, inserting new data timestamps or version numbers. When reading, the timestamp or version number is checked to retrieve the latest data, converting random writes to sequential writes. However, without a mechanism to handle expired data, this update mechanism leads to a large amount of outdated data. Furthermore, reading data requires reading multiple files and comparing timestamps, resulting in lower read performance and space utilization compared to in-site updates, thus failing to achieve real-time updates.

[0007] A complete column-oriented index currently includes metadata and data. When designing its storage structure, the column-oriented index stores the metadata and data structures as files on disk. During index creation, the column-oriented index is saved to external storage multiple times; each save is called a column-oriented index segment, which includes the complete column-oriented index structure. Typically, when creating a large document set, multiple column-oriented index segments are formed. For the same column, its data is distributed across multiple segments. To avoid this, the common practice is to merge column-oriented index segments, reloading all segments into memory and merging them into a larger column-oriented index segment. This process is as follows... Figure 1As shown. This merging method can improve the performance of index queries, but merging all columnstore index segments at the end consumes a lot of time and puts a great deal of pressure on memory. Currently, search engine libraries such as Doc Values ​​perform multiple rounds of columnstore index segment merging during index creation, using a multi-threaded approach to merge some segments during the columnstore index creation process, saving time in creating the columnstore index. However, this causes a significant amplification of the index size, and the final round of segment merging still consumes some of the time required to create the columnstore index.

[0008] Sorting is an indispensable part of data analysis. Sorting data helps to quickly and intuitively display and better understand it, facilitates the organization and retrieval of needed data, and ultimately contributes to making more effective decisions. For columnar storage systems, traditional sorting methods, such as Doc Values ​​and related storage engines, utilize external sorting, reading large amounts of data from disk into memory for sorting and merging the results. This requires significant data loading time and has high memory requirements. Insufficient memory can lead to frequent I / O and excessive CPU resource consumption, resulting in performance degradation or even system crashes.

[0009] Columnar storage can be divided into indicator columns and dimension columns based on the characteristics of each column. Indicator columns are numeric types, suitable for scenarios that are frequently updated but do not require building sorted indexes, while dimension columns are generally string types and are often used for grouping and sorting operations, but are generally not updated. However, current storage engines do not process each column separately according to its characteristics, which is not conducive to improving the performance of columnar storage engines.

[0010] 1. Currently, the following are solutions for fast querying, updating, and sorting based on columnar storage technology:

[0011] 1) In-memory databases offer high read and write efficiency, but their storage capacity is limited, and high-capacity memory is expensive, thus limiting their widespread adoption. Columnar storage databases based on external storage devices can handle massive data volumes. When data is updated, the Doc Values ​​index uses Recurrent Computation (RCU) for updates. However, in real-time update scenarios, frequent updates lead to a large amount of duplicate data writing, resulting in write amplification.

[0012] 2) Search engine frameworks like DocValues ​​store columnar indexes on external storage devices in index segment structure, following the Doc Number order. This process creates multiple index segments. During index creation, existing columnar index segments are merged to form larger segments until the segment size reaches a set threshold. This merging process can lead to excessively long creation times. While multi-round segment merging, using multi-threading during columnar index creation, saves time compared to single-round merging, it also results in the same data being written multiple times, causing write amplification of the columnar index.

[0013] 3) When performing equality sorting queries, search engine frameworks such as Lucene often need to load all index segments on the disk into memory for traversal and sorting. This is very unfriendly to scenarios with insufficient memory, causing great pressure and very slow query speed.

[0014] 2. Non-volatile memory cannot directly improve the performance of columnar organization.

[0015] Non-Volatile Memory (NVM), a new type of hardware storage medium, also known as non-volatile memory, possesses physical characteristics similar to RAM, such as powerful read / write performance (its random read / write speed can reach several GB / s), wear resistance, and the ability to retain data even when power is off, just like hard disks. Many scholars consider it the future of storage devices. Currently, commercially available NVM devices—such as Optane Memory—possess the byte-addressing characteristics of DRAM and can be directly manipulated via CPU instructions, while also retaining the non-volatility of hard disks, and at a lower cost than DRAM. If NVM matures further, it will be of great significance to database products. For applications with I / O bottlenecks, directly applying NVM can significantly improve performance. However, replacing block devices like hard disks with NVM as the storage device for columnar indexes does not speed up the creation and retrieval of columnar storage. This is mainly because current storage architectures are designed for block devices like hard disks and are not optimized for the physical characteristics of NVM. Summary of the Invention

[0016] The purpose of this invention is to propose a columnar storage indexing method and apparatus based on non-volatile memory (NVM). This method leverages the powerful random access performance of NVM and its advantages such as byte addressing, non-volatility, and low cost. The columnar storage structure is redesigned. For scenarios where index columns are updated in real-time, a two-stage batch update is used. Metadata merging is used during index creation instead of traditional columnar storage index merging, thereby reducing memory usage and read / write amplification without compromising read / write performance. For low-cardinality columns, an inverted index is created, significantly improving equality query performance while substantially reducing memory usage.

[0017] To achieve the above objectives, the present invention adopts the following technical solution:

[0018] A columnar storage indexing method based on non-volatile memory includes the following steps:

[0019] Create a cache pool NVMPOOL in the non-volatile memory NVM;

[0020] In NVMPOOL, construct the Main structure and Delta structure. The Main structure includes multiple columnar index segments, each segment containing metadata information and data information. The data information is stored in several data segments of the DVD file, and the metadata information records the address of each data segment and logically connects the data segments. The metadata information is stored in the DVM file. The Delta structure includes two Delta files.

[0021] Bloom filters are built in memory for the two Delta files of the Delta structure. The Bloom filters are used to query and determine whether the data in the indicator column has been updated.

[0022] Each time the data in the indicator column is updated, the update request is parsed in memory to determine the segment where the update request is located. The updated data is written to a Delta file in the Delta structure of the segment. When the Delta file is full, the process switches to another Delta file. At the same time, the relevant column data information in the DVD file is located by loading the DVM file and loaded into memory. The full Delta file is merged to update the data. The updated data is then written to the DVD file of the NVM and the Bloom Filter corresponding to the Delta file is released.

[0023] When data is written to the NVM DVD file, an inverted index is built. This inverted index consists of a dictionary and an inverted list. The dictionary resides in memory and records the terms after deduplication and sorting of the dimension columns. Each term corresponds to a sorted inverted list.

[0024] Preferably, each segment is divided using a document number.

[0025] Preferably, after determining the segment where the update request is located, the row offset address of the update request and the Doc Number corresponding to the segment are first calculated through the DVM file; after writing the update data into a Delta file in the Delta structure of the segment, the position of the Doc Number in the Bloom Filter corresponding to the Delta file is marked.

[0026] Preferably, an inverted linked list records all Doc Numbers for the term and sorts them according to the Doc Numbers.

[0027] Preferably, the keys in the Delta structure record the Row-offset.

[0028] Preferably, the data segment includes a segment identifier, a segment number, columnar data, and a segment tail identifier; wherein, the segment identifier is used to record the version number; the segment number is used to distinguish data segments in the entire NVMPOOL; the columnar data is distinguished according to the type of the field, if the field is integer data, it is recorded using fixed bytes; if the field is string data, the data length is set before each field value; the segment tail identifier is used to determine the end of the data segment.

[0029] Preferably, the column index of the same field is stored in all data segments. The column name and type that are the same in each segment are used as the key. The metadata information of the same key in each segment is merged. After merging, there is only one metadata information in each NVMPOOL.

[0030] Preferably, the DVM file structure includes segment identifiers, Doc Values, and segment identifiers. Doc Values ​​include column name, column type, column number, number of segments, and column index. The column index includes location information and data length. The segment identifier is used to record the segment number and number of columns. The number of segments is used to represent the number of merged DVMs. The Doc Number in the merged column index is the sum of the Doc Numbers of each merged segment. The location information and data length are used to record the starting offset and size of the same key in each segment.

[0031] Preferably, when querying multiple terms simultaneously, it is only necessary to find the inverted linked list of the corresponding term in sequence and read the inverted linked list corresponding to the term in each data segment into memory.

[0032] A columnar storage indexing device based on non-volatile memory, comprising:

[0033] The non-volatile memory (NVM) uses columnar data storage and includes a cache pool (NVMPOOL). The NVMPOOL stores the Main structure and the Delta structure. The Main structure comprises multiple columnar index segments, each containing metadata and data information. The data information is stored within several data segments of the DVD file, while the metadata information records the addresses of each data segment and logically connects them. The metadata information is stored in the DVM file. The Delta structure includes two Delta files.

[0034] The memory contains two Delta files with a Delta structure, each with a Bloom filter. The Bloom filters are used to query and determine whether the data in the indicator columns has been updated.

[0035] Each time the data in an indicator column is updated, the update request is parsed in memory to determine the segment where the update request belongs. The updated data is then written to a Delta file in the Delta structure of the segment. When the Delta file is full, the process switches to another Delta file. At the same time, the relevant column data information in the DVD file is located by loading the DVM file, and this data information is loaded into memory to merge the full Delta file for data update. The updated data is then written to the NVM DVD file, and the Bloom Filter corresponding to the Delta file is released. When the data is written to the NVM DVD file, an inverted index is built. This inverted index consists of a dictionary and an inverted list. The dictionary resides in memory and records the deduplicated and sorted terms of the dimension columns. Each term corresponds to a sorted inverted list.

[0036] Advantages of this invention:

[0037] 1) For frequently updated indicator columns, to achieve real-time updates, this invention employs a two-stage batch update strategy within segments, caches update requests again in NVMPOOL, and reduces random accesses by building a Bloom filter in memory, thereby ensuring query performance. This invention achieves real-time updates while maintaining query performance, reducing update creation time and write amplification caused by frequent updates.

[0038] 2) For the indicator column, this invention redesigns the column-oriented storage metadata information and data structure for inter-segment data. During index creation, leveraging the powerful random read performance of NVM, metadata information merging replaces complete column-oriented index merging; data segments are only logically merged. Compared to existing column-oriented index storage structures, this invention reduces the time required to create column-oriented indexes and minimizes write amplification during multiple rounds of segment merging. This invention distributes data information from the same column across NVMPOOLs; due to NVM's powerful random read / write performance, loading data from multiple locations does not cause significant performance loss.

[0039] 3) For low cardinality dimension columns, this invention utilizes an inverted index for sorting to achieve equality queries. During reading, it only needs to traverse the dictionary and retrieve the inverted list corresponding to the term. Compared to existing sorted index structures, this invention significantly reduces memory usage and improves equality query performance. Attached Figure Description

[0040] Figure 1 This is a diagram illustrating the merging of Doc Values ​​index segments.

[0041] Figure 2 This is a diagram of the NVMPOOL design and data segment structure.

[0042] Figure 3 This is a flowchart of a two-stage data update strategy.

[0043] Figure 4 This is a diagram showing the structure of merged DVM metadata information.

[0044] Figure 5 This is a flowchart of the metadata information merging process.

[0045] Figure 6 This is a flowchart of the inverted index construction process.

[0046] Figure 7 This is a flowchart of the equality lookup process. Detailed Implementation

[0047] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

[0048] For columnar storage engines, index columns are suitable for scenarios with frequent updates, while dimension columns are often used for grouping and sorting operations. This invention addresses these two scenarios by leveraging the powerful random read / write performance and byte addressing capabilities of NVM to improve upon traditional columnar storage structures. Traditional columnar storage engines mostly perform in-place updates and typically employ RCU (Read-Copy-Update) to avoid lock contention. However, this update strategy can cause severe read / write amplification in frequent update scenarios. This invention achieves real-time updates by using two-stage batch updates within segments and employing Bloom filters. Traditional columnar indexes store metadata and data information separately using different file types, then merge this information into a columnar index segment, resulting in write amplification and lengthy segment merging time. This invention establishes a cache pool, NVMPOOL, within NVM to store the complete columnar index. When creating inter-segment indexes, only the stored metadata information is merged; data segments containing data information are only logically merged. Traditional sorting methods use external sorting, which reads all the data into memory for sorting. If there are multiple columnar storage segments, the results also need to be merged in memory, resulting in a large memory footprint. Furthermore, the frequent I / O and excessive CPU consumption caused by external sorting lead to a decrease in system efficiency. This invention utilizes an inverted index to reduce memory footprint and improve the efficiency of equality queries.

[0049] The column-oriented indexing method based on non-volatile memory proposed in this invention is described in detail below:

[0050] (I) Two-stage data update strategy for indicator columns

[0051] To better design columnar indexes, avoid the impact of general file systems, and utilize the byte addressing characteristics of NVM, this invention creates a cache pool (NVMPOOL) in NVM, where offsets can be used as addresses. Figure 2This paper introduces the design and data segment structure of NVMPOOL. NVMPOOL stores two file structures: metadata information and data segments containing data information. The metadata information is mainly responsible for locating the data information within a data segment. The metadata information designed in this invention not only stores information about each data segment but also connects them, ensuring logical order. A data segment mainly consists of four modules: segment identifier, Segment_ID (segment number), columnar data, and segment tail identifier. The segment identifier mainly records information such as the version number, and the segment number is used to distinguish data segments within the entire NVMPOOL. For the columnar data module, the data is distinguished based on the field type. If it is integer data, it is recorded using fixed bytes, laying the foundation for byte addressing using NVM. If it is string data, to avoid excessive space waste, a data length is set before each field value to ensure the correctness of the value. The end of a segment can be determined by the segment tail identifier.

[0052] Columnar storage updates have always been a weakness of columnar storage engines. Therefore, this invention addresses this by performing phased batch updates on index column data, transforming random writes into sequential writes, thereby reducing random I / O and write amplification. This invention employs a two-tier architecture: a Main structure and a Delta structure, both stored in NVMPOOL. The Main structure primarily contains multiple columnar storage index segments, separated by document numbers (Doc Numbers). Each segment's segment number is unique within that segment, ensuring that each update corresponds to only one partition (Segment), avoiding rewriting data across all areas and significantly improving performance. This invention performs two-stage data updates on each Segment. Each Segment contains metadata and data information. The Delta structure corresponds to two files, Delta1 and Delta2. When one Delta file is full, data merging and updating are completed through file switching and background merging. The keys in the Delta structure are not primary keys, but rather record the row offset address that is being updated. This eliminates the need for string comparison when locating Delta content, thus greatly speeding up the location process.

[0053] When performing a query, the two-stage update strategy of this invention needs to query the Delta structure and the Main structure because it is uncertain whether the value corresponding to the query request has been updated, resulting in three random accesses. In order to reduce the number of unnecessary random accesses, this invention constructs Bloom filters for the two Delta files in memory. Figure 3The steps demonstrate the batch update and merging process, where steps (1) to (2) are two-stage batch update processes, and steps (3) to (6) are the merging process of Delta structure and Main structure. (1) First, a Bloom filter is used on the updated data to determine whether the value has been updated and the Delta file where the update is located during the query; (2) The Row-offset of the updated value is calculated through the DVM file of the metadata information, and the Row-offset and the updated value are written into a Delta file, and the position of the Doc Number in the Bloom Filter corresponding to the Delta file is marked as 1 (1 represents the position is occupied, and 0 represents the position is not occupied). If Delta reaches the threshold, switch Delta files and merge Delta files that have reached the threshold with files in the Main structure: (3) Load the DVM file and locate the data information in the DVD file through the information in the DVM file. The DVD file is used to record data information. This step aims to locate the data information through metadata information; (4) Read the data information of the relevant columns in the DVD file according to the DVM file; (5) Load the data information of the relevant columns in the DVD file into memory and update the data in memory; (6) Write the updated data back to the DVD file.

[0054] (II) Logical Merging Method for Data Segments Oriented to Index Columns

[0055] The segment merging process of column-store indexes consumes a significant amount of time and puts considerable pressure on systems with insufficient memory space, resulting in severe write amplification. Therefore, this invention utilizes the characteristics of NVM to design a new column-store index structure for index columns, leveraging NVM's powerful random read / write performance to reduce the merging of column-store index segments during the creation of column-store indexes.

[0056] Before merging, each data segment is independent of the others. Column indexes for the same field are stored in all data segments. This invention uses a DVM metadata information structure to maintain all data information in each column. It utilizes the existence of columns with the same name and type in each segment as keys, merging the metadata information for the same key in each segment. After merging, each NVMPOOL contains only one piece of metadata information. The DVM metadata information storage structure is as follows: Figure 4As shown, the segment identifier mainly records the segment number, column number, etc. The segment number represents the number of merged DVMs. The number of rows in the merged column index (i.e., the number of Doc Numbers) is the sum of the number of rows in each merged segment (i.e., the number of Doc Numbers). The position information and data length record the starting offset and size of the same key in each index segment. The metadata information connects all the data segments in NVMPOOL into a complete columnar index.

[0057] Considering that writing data with too small a granularity can lead to performance degradation, the segments in this invention are designed in MB-level units. The NVM metadata information structure stores the addresses of each data segment and connects them; therefore, each data segment corresponds to a portion of the metadata information. When a certain number of data segments are reached, the metadata information is updated. A diagram illustrating metadata merging is shown below. Figure 5 (1) When the cache in memory reaches a certain size, the columnar storage index segment is saved as metadata information (DVM file) and data information (DVD file). At this time, the metadata information corresponds one-to-one with the data information, and the metadata information saves the address information in the data information; (2) When the amount of metadata information written reaches a certain quantity, it is reloaded into memory; (3) A new metadata information file is regenerated, so that the data segment becomes logically merged.

[0058] (III) Index Optimization Based on Dimension Columns in Inverted Indexes

[0059] One important application of columnar storage is equality sorting of low-cardinality columns. Traditional columnar storage sorting methods read all data into memory for equality sorting, which not only consumes a large amount of memory but also requires frequent I / O and merging using external sorting, consuming excessive CPU resources. Therefore, this invention builds an inverted index when writing data, thereby improving equality query performance while reducing memory usage. Furthermore, when building the inverted index, the number of documents under each term is recorded simultaneously. When performing statistical operations, there is no need to read the data into memory; data statistics can be performed in memory. The inverted index construction flowchart is shown below. Figure 6 .

[0060] The inverted index in this invention is mainly divided into two parts: a dictionary and an inverted linked list. The dictionary records the terms of the dimension column after deduplication and sorting. Because the dictionary occupies a small space, it is kept in memory. At the same time, each term corresponds to a sorted inverted linked list, which records all Doc Numbers containing the term and sorts them according to the Doc Number.

[0061] When querying multiple terms simultaneously, since the dictionary is ordered, it is only necessary to find the inverted linked list of the corresponding terms sequentially. Furthermore, since the inverted linked list segment corresponding to each term is ordered, it is only necessary to read the linked list corresponding to that term in each segment into memory, without having to read all the data from disk into memory for sorting. Figure 7 This shows the results of an equality query performed using an inverted index.

[0062] This invention significantly reduces memory usage, greatly decreases the number of interactions between memory and disk, and improves the performance of equality queries. Furthermore, it avoids data transfer overhead by utilizing NVM byte addressing features and low-latency loads and stores instructions.

[0063] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A columnar storage indexing method based on non-volatile memory, characterized in that, Includes the following steps: Create a cache pool NVMPOOL in the non-volatile memory NVM; In NVMPOOL, construct the Main structure and Delta structure. The Main structure includes multiple column-oriented index segments, each containing metadata and data information. The data information is stored in several data segments of the DVD file, and the metadata information records the addresses of each data segment and logically connects the data segments. The metadata information is stored in the DVM file. The Delta structure includes two Delta files, each used to store the updated data of the index column in the corresponding Segment. The updated data includes the row offset address and its corresponding update value. Bloom filters are built in memory for the two Delta files of the Delta structure. The Bloom filters are used to query and determine whether the data in the indicator column has been updated. Each time the data in the indicator column is updated, the update request is parsed in memory to determine the segment where the update request is located. The updated data is written to a Delta file in the Delta structure of the segment. When the Delta file is full, the process switches to another Delta file. At the same time, the relevant column data information in the DVD file is located by loading the DVM file and loaded into memory. The full Delta file is merged to update the data. The updated data is then written to the DVD file of the NVM and the Bloom Filter corresponding to the Delta file is released. When data is written to the NVM DVD file, an inverted index is built. This inverted index consists of a dictionary and an inverted list. The dictionary resides in memory and records the terms after deduplication and sorting of the dimension columns. Each term corresponds to a sorted inverted list.

2. The method as described in claim 1, characterized in that, Each segment is divided using a document number.

3. The method as described in claim 2, characterized in that, After determining the segment where the update request is located, the row offset address of the update request and the Doc Number corresponding to the segment are first calculated through the DVM file. After writing the update data into a Delta file in the Delta structure of the segment, the position of the Doc Number in the Bloom Filter corresponding to the Delta file is marked.

4. The method as described in claim 1 or 2, characterized in that, An inverted linked list records all Doc Numbers for this term and is sorted by Doc Number.

5. The method as described in claim 1, characterized in that, The keys in the Delta structure record the Row-offset.

6. The method as described in claim 1, characterized in that, A data segment includes a segment identifier, a segment number, columnar data, and a segment tail identifier. The segment identifier is used to record the version number. The segment number is used to distinguish data segments within the entire NVMPOOL. The columnar data is distinguished according to the field type. If the field is an integer, it is recorded using fixed bytes. If the field is a string, the data length is set before each field value. The segment tail identifier is used to determine the end of the data segment.

7. The method as described in claim 1, characterized in that, The column index of the same field is stored in all data segments. The column name and type that are the same in each segment are used as the key. The metadata information of the same key in each segment is merged. After merging, there is only one metadata information in each NVMPOOL.

8. The method as described in claim 1, characterized in that, The DVM file structure includes segment identifiers, Doc Values, and segment identifiers. Doc Values ​​include column names, column types, column numbers, number of segments, and column indexes. The column indexes include location information and data length. The segment identifiers record the segment number and number of columns, the number of segments indicates the number of merged DVMs, and the Doc Number in the merged column index is the sum of the Doc Numbers of each merged segment. The location information and data length record the starting offset and size of the same key in each segment.

9. The method as described in claim 1, characterized in that, When querying multiple terms simultaneously, it is only necessary to find the inverted linked list of the corresponding terms in sequence and read the inverted linked list corresponding to the term in each data segment into memory.

10. A columnar storage indexing device based on non-volatile memory, characterized in that, include: The non-volatile memory (NVM) uses columnar data storage and includes a cache pool (NVMPOOL). The NVMPOOL stores a Main structure and a Delta structure. The Main structure comprises multiple columnar index segments, each containing metadata and data information. The data information is stored within several data segments of the DVD file, while the metadata information records the addresses of each data segment and logically connects them. This metadata information is stored in the DVM file. The Delta structure includes two Delta files, each storing updated data for the corresponding index column in the Segment. The updated data includes the row offset address and its corresponding update value. The memory contains two Delta files with a Delta structure, each with a Bloom filter. The Bloom filters are used to query and determine whether the data in the indicator columns has been updated. Each time the data in an indicator column is updated, the update request is parsed in memory to determine the segment where the update request belongs. The updated data is then written to a Delta file in the Delta structure of the segment. When the Delta file is full, the process switches to another Delta file. At the same time, the relevant column data information in the DVD file is located by loading the DVM file, and this data information is loaded into memory to merge the full Delta file for data update. The updated data is then written to the NVM DVD file, and the Bloom Filter corresponding to the Delta file is released. When the data is written to the NVM DVD file, an inverted index is built. This inverted index consists of a dictionary and an inverted list. The dictionary resides in memory and records the deduplicated and sorted terms of the dimension columns. Each term corresponds to a sorted inverted list.

Images