An inverted index storage apparatus based on non-volatile memory

Abstract

The application discloses an inverted index storage device based on a non-volatile memory, which comprises at least one virtual section used for storing basic information of the inverted index, and an NVM dictionary used for maintaining the basic information in each virtual section based on a virtual section dynamic address array contained in each word item in the NVM dictionary, wherein the word item and the basic information are obtained through an index section of separated data. The application uses dictionary merging to replace traditional inverted index section merging when creating the inverted index, accelerates the creation of the inverted index, and greatly reduces the write amplification problem caused by the creation of the inverted index.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to an inverted index storage device based on non-volatile memory. Background Technology

[0002] With the rapid development of the information age and mobile internet, data has experienced explosive growth. Applications such as Weibo, Zhihu, and various news sites generate massive amounts of unstructured text data daily. Inverted indexes are a crucial underlying structure for text retrieval, helping users quickly locate the document ID of a query within massive amounts of text data and sort the results using appropriate scoring methods. While inverted indexes can significantly accelerate full-text search efficiency, their creation is an extremely time-consuming task, requiring the simultaneous traversal of all documents, word segmentation, creation of the inverted table, and the integration of positional information.

[0003] Creating an inverted index is time-consuming. To prevent unexpected program exits, the inverted index in memory is saved to external storage devices such as disks and SSDs during the index creation process. However, external storage devices have weak random access performance. Currently, to avoid random reads and writes during queries, the inverted index stores the inverted table of a certain term in a contiguous block of memory. This means that if the entire index cannot be written to disk at once, the inverted table of the same term will be stored in multiple non-contiguous external storage spaces, increasing query latency.

[0004] A complete inverted index includes a dictionary, an inverted record table, a skip list, and position information. Currently, inverted indexes store the dictionary and inverted table as files on disk during storage design. During index creation, the inverted index is saved to external storage multiple times; each saved data segment can be called an inverted index segment, containing the complete inverted index structure. Typically, when creating a large document set, multiple inverted index segments are formed. For a single term, the complete inverted table may reside in multiple segments. To avoid this, the common practice is to merge inverted index segments, reloading all segments into memory and merging them into a larger inverted index segment. This process is as follows: Figure 1 As shown.

[0005] This merging method can improve the performance of index queries, but merging all inverted index segments at the end consumes a lot of time. Currently, search engine frameworks such as Lucene perform multiple rounds of inverted index segment merging during index creation, using a multi-threaded approach to merge partial segments during the inverted index creation process, thus saving time in creating the inverted index.

[0006] Specifically, existing technologies have the following drawbacks:

[0007] 1. Write amplification caused by multiple rounds of inverted index segment merging:

[0008] Search engine frameworks like Lucene store inverted index information in dictionary order as index segments on external storage during index creation. This process generates multiple index segments. During index creation, existing inverted index segments are merged to form larger segments until the segment size reaches a set threshold. While multi-round segment merging, performed using multi-threading during inverted index creation, saves time compared to single-round merging, it also leads to the same data being written multiple times, causing inverted index write amplification.

[0009] 2. Non-volatile memory cannot directly improve the performance of inverted indexes:

[0010] Non-volatile memory (NVM), as a relatively new storage device, possesses physical characteristics similar to RAM, such as powerful read / write performance (its random read / write performance can reach GB / s levels), byte addressability, and wear resistance. Like hard disks, it retains data even when power is off, leading many scholars to consider it the future of storage devices. Currently, commercially available NVM devices—such as Optane Memory—can significantly improve the performance of applications with I / O bottlenecks. However, replacing block devices like hard disks with NVM as the storage device for inverted indexes does not speed up the creation and retrieval of inverted indexes. This is primarily because current storage architectures are designed for block devices like hard disks and are not optimized for the physical characteristics of NVMs.

[0011] In conclusion, given the massive amount of data updates every day, accelerating the creation of inverted indexes has become an urgent problem to be solved. Summary of the Invention

[0012] The purpose of this invention is to design an inverted index storage device based on non-volatile memory. This storage device utilizes the powerful random read / write performance and byte addressing advantages of NVM to redesign the dictionary and inverted index segment structure. When creating the inverted index, dictionary merging is used instead of traditional inverted index segment merging, which speeds up the creation of the inverted index and greatly reduces the write amplification problem caused by the creation of the inverted index.

[0013] To achieve the above objectives, the present invention adopts the following technical solution:

[0014] An inverted index storage device based on non-volatile memory, comprising:

[0015] At least one virtual segment is used to store basic information about the inverted index;

[0016] An NVM dictionary is used to maintain the basic information in each virtual segment based on the virtual segment dynamic address array contained in each term in the NVM dictionary;

[0017] The term and inverted index basic information are obtained by segmenting the document into words and building an inverted index in memory. The inverted index basic information includes: inverted record table, word position information and skip list.

[0018] Furthermore, the structure of the virtual segment includes: a segment header, a data segment, and a segment footer, wherein,

[0019] The data segment is used to store basic information about the inverted index;

[0020] The segment tail includes: redundant space and segment tail identifier;

[0021] The segment header includes: segment identifier, inverted index, data length, address range, and redundancy space size, wherein,

[0022] The inverted index is used to record the number of terms corresponding to the basic information of the inverted index in the current virtual segment;

[0023] The data length is used to record the number of bytes in the current virtual segment;

[0024] The address range is used to record the start and end addresses of the current virtual segment in the cache pool of the non-volatile memory;

[0025] The redundant space size is used to record the size of the redundant space in the current virtual segment.

[0026] Furthermore, the structure of the NVM dictionary includes: dictionary version, number of terms, skip list span, and term content, characterized in that the term content includes:

[0027] PreLen is used to record the number of times the prefix of the term is the same as that of the previous term;

[0028] Suffix is ​​used to indicate a word suffix;

[0029] FieldNum is used to represent the number of fields in which the term appears;

[0030] Count indicates the total number of times the term appears;

[0031] The DocFreq array represents the total number of documents in which this term appears;

[0032] The Skips array is used to record the address and document number of the corresponding skip list in each virtual segment, wherein the skip list is stored in the data segment;

[0033] The Poss array is used to record the address of the corresponding location information in each virtual segment.

[0034] A method for creating and incrementally updating the above-mentioned inverted index includes the following steps:

[0035] After word segmentation of the document, an inverted index structure is built in memory. The inverted index structure includes basic information about terms and inverted indexes.

[0036] If the space occupied by the inverted index structure in memory reaches the set first threshold, a virtual segment is created in the non-volatile memory cache pool file, and the basic information of the inverted index is stored in the virtual segment;

[0037] A single-segment dictionary corresponding to a single virtual segment is created in memory, and the single-segment dictionary is saved to a non-volatile memory cache pool. The dictionary structure of the single-segment dictionary is the dictionary structure commonly used in traditional inverted indexes.

[0038] The document set is continuously processed, and new virtual segments are continuously generated in the non-volatile memory cache pool file. The corresponding single-segment dictionary is appended to the non-volatile memory cache pool.

[0039] When the number of single-segment dictionaries reaches the second threshold, the existing single-segment dictionaries are merged, and the merged dictionary structure is the NVM dictionary structure.

[0040] When document set processing is complete, all dictionaries in non-volatile memory (including single-segment dictionaries or NVM dictionaries that have not yet been merged) are merged into a single NVM dictionary containing all terms in the entire document set. At this point, the inverted index creation or incremental update is complete.

[0041] Furthermore, during the inverted index merging process, only the dictionary is merged, and a virtual segment is created in the non-volatile memory cache pool file to store the basic information of the inverted index. It also includes: determining whether the basic information of the inverted index of the term is written to the non-volatile memory based on the length of the inverted table in memory.

[0042] Compare the length with a third threshold:

[0043] If the length is not less than the third threshold, then the basic information of the inverted index is stored in non-volatile memory;

[0044] If the length is less than the third threshold, the basic information of the inverted index is delayed until the accumulated length of the inverted record table is not less than the third threshold or the number of delayed writes reaches the set value, and then the basic information of the inverted index is stored.

[0045] A background merging method for low-frequency and scattered inverted index information in the aforementioned inverted index storage device, comprising the following steps:

[0046] Find the total length of the inverted index and the number of virtual segments containing the inverted index for each term in the NVM dictionary;

[0047] Based on the total length of the inverted index and the number of virtual segments, calculate the average length p of the inverted index stored in each virtual segment;

[0048] Compare the length p of the inverted index table with the fourth threshold:

[0049] If the average inverted index length p is not less than the fourth threshold, then the inverted index of that term will not be processed.

[0050] If the average inverted index length p is less than the fourth threshold, the inverted index of the term is merged, and a virtual segment is selected or created for storage.

[0051] A storage medium storing a computer program, wherein the computer program is configured to execute any of the methods described above when run.

[0052] An electronic device includes a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform any of the methods described above.

[0053] This invention proposes a method to accelerate inverted index creation using NVM. It redesigns the dictionary and inverted record table structure of the inverted index and leverages the powerful random read performance of NVM by merging the dictionary instead of merging all inverted records during index creation. Compared with existing technologies, this invention has at least the following advantages:

[0054] 1. This invention significantly reduces write amplification during the creation of inverted indexes and shortens the index creation time;

[0055] 2. Due to NVM's powerful random read and write performance, multiple random reads caused by the scattered storage of inverted index data of the same term in NVMPOOL will hardly cause performance loss. Furthermore, the significantly shortened merging time greatly improves the speed of incremental updates. Attached Figure Description

[0056] Figure 1 The merging process of inverted index segments in existing technologies.

[0057] Figure 2 NVMPOOL Design and Virtual Segment Structure.

[0058] Figure 3 NVM dictionary structure diagram.

[0059] Figure 4 Skip list structure design.

[0060] Figure 5 Dictionary merging design.

[0061] Figure 6 Inverted records that need to be merged.

[0062] Figure 7 Low-frequency inverted index delayed write.

[0063] Figure 8 Inverted indexes are merged in the background.

[0064] Figure 9 Index backend migration.

[0065] Figure 10 Comparison of inverted index creation times.

[0066] Figure 11 Incremental updates to the inverted index.

[0067] Figure 12 Comparison of query latency for different word frequencies.

[0068] Figure 13 The amount of data written to create the index.

[0069] Figure 14 Changes in dictionary size after merging fine-grained inverted indexes.

[0070] Figure 15 Changes in query latency for low-frequency terms after merging fine-grained inverted indexes.

[0071] Figure 16 Comparison of query latency across different media before and after index merging. Detailed Implementation

[0072] To make the above features and advantages of the present invention more apparent and understandable, the technical solution of the present invention will be further described below through specific embodiments.

[0073] This invention improves the storage structure of traditional inverted indexes by leveraging the powerful random read / write performance and byte addressing capabilities of NVM. Traditional inverted indexes store dictionary, inverted record table, and position information separately using different file types, then merge this information into an inverted index segment. During index creation, segment merging occurs, causing write amplification and consuming significant time. This invention establishes a cache pool file, NVMPOOL, to store the complete inverted index and uses the NVM transactional read / write interface to avoid consistency issues caused by writes regarding dictionary length and inverted index segment length. During index creation, the cached inverted index is written to NVMPOOL as virtual segments without merging. The improvements made by this invention based on NVM are as follows:

[0074] (I) Virtual Segment Structure and NVM Dictionary Structure

[0075] like Figure 2 As shown, this invention stores a dictionary and several virtual segments in the NVMPOOL. The virtual segments store the inverted index (excluding the dictionary). The NVM dictionary designed in this invention not only stores terms but also connects the virtual segments. The virtual segments mainly store the inverted index, position information, etc. The structure of the NVMPOOL and the virtual segments is as follows: Figure 1 As shown.

[0076] This invention uses virtual segments to store inverted index information. A complete inverted index consists of multiple virtual segments, but these segments are independent of each other and do not require index segment merging. A virtual segment mainly consists of three modules: a segment header, data, and a segment footer. The segment header primarily records the virtual segment's metadata, including the segment identifier, inverted index number, data length, address range, and redundancy space size. The segment identifier distinguishes virtual segments within the entire NVMPOOL, the inverted index number records the number of terms in the current virtual segment, the data length represents the number of bytes in the virtual segment's data, the address range records the start and end addresses of this segment in the NVMPOOL, and the redundancy space size records the amount of unused space in the virtual segment. The virtual segment data stores basic information about the inverted index, including the inverted record table, position information, and forward index information. The segment footer contains two parts: redundancy space and a segment footer identifier. The reason for reserving some space is that after the index is built, incremental updates to the inverted index may be performed; therefore, some space is reserved for preparation. When the amount of new data added to a virtual segment exceeds the reserved space, if there is space after the virtual segment, the end of the virtual segment is moved and the beginning of the virtual segment is modified. If there is insufficient space within the virtual segment, a new space is allocated, and the original virtual segment is migrated. An inverted index for a term may be stored in multiple virtual segments. This invention connects all virtual segments by establishing an address array in the dictionary. The structure of the NVM dictionary is as follows: Figure 3 As shown.

[0077] This invention uses an NVM dictionary structure to maintain the entire inverted index table. A dynamic address array is added to the normal dictionary to store the addresses of each term in different virtual segments. The dictionary connects all virtual segments in the NVMPOOL into a complete inverted index. For example... Figure 3As shown, each NVMPOOL contains only one dictionary. The dictionary version indicates the number of times the dictionary was updated during dictionary creation. The term count indicates the number of words contained in the dictionary. SkipInteval indicates the span of the skip list. The basic unit in the dictionary is TermInfo. In TermInfo, PreLen indicates the number of times the term shares the same prefix with the previous term, and Suffix indicates the term suffix. FieldNum indicates the number of fields in which the term appears, Count indicates the total number of times the term appears, and DocFreq indicates the total number of documents in which the term appears. In a normal inverted index dictionary, skip lists, position information, and the offset of the inverted record table are usually recorded. Since the NVM inverted record table is stored in a distributed manner, multiple addresses are also needed to store it in the dictionary. Freqs is an array, and each item in the array represents the address of the virtual segment in which the term appears in the inverted record table. For example, if a word appears in 5 virtual segments, then the length of Freqs is 5. Poss is an array, and each item represents the address of the corresponding position information in each virtual segment.

[0078] Skip lists can significantly speed up logical operations (intersection operations) of inverted indexes. Due to changes in the underlying design of the NVM inverted index, this invention also redesigns the skip list, such as... Figure 4 As shown. Each virtual segment contains an inverted index. When the length of the inverted index reaches a certain threshold, a skip list is created for it, and the skip list is also saved in the virtual segment along with the inverted index. Simultaneously, the NVM dictionary records the address and document number of the corresponding skip list in each virtual segment, stored in the Skips array. Furthermore, when the length of the Skips array reaches a certain threshold, multi-level skip lists are created within the Skips array.

[0079] (ii) Dictionary merging replaces merging of the entire inverted index segment

[0080] The NVM dictionary structure designed in this invention stores the addresses of each virtual segment and connects them. When writing a new index segment, such as... Figure 5 As shown, new virtual segment addresses need to be added to the NVM dictionary. This invention selects a dictionary merging strategy: during the creation of the inverted index, as inverted data is written to each virtual segment, the dictionary of a single virtual segment (referred to as a single-segment dictionary) is appended to the NVMPOOL. When the number of single-segment dictionaries reaches a set threshold, the single-segment dictionaries are merged to obtain the NVM dictionary, as shown below. Figure 3As shown, multiple NVM dictionaries are generated during the entire document set processing. After the document set processing is complete, the entire dictionary is merged, ultimately retaining only one NVM dictionary containing all terms. By using the list of virtual segment addresses recorded in each term, complete inverted index information can be directly loaded from multiple virtual segments during querying, leveraging NVM's powerful random read / write performance to achieve fast queries.

[0081] (III) Fine-grained inverted record merging method

[0082] For a given term, its inverted index is typically stored contiguously on disk, requiring only one address to be stored in the dictionary. However, the inverted index proposed in this invention stores each inverted record in a distributed manner within an NVM (Network Virtual Machine). Therefore, the NVM dictionary designed in this invention stores multiple addresses for each term. For some overly distributed inverted indexes, the dictionary would need to store each address, resulting in an excessively large dictionary. For example... Figure 6 As shown, the inverted index of a term includes 5 documents, but they are stored in 5 segments. Therefore, the dictionary also needs 5 segment addresses, resulting in unnecessary space waste. Furthermore, the XP Buffer in NVM has a size of 256B per row. Some low-frequency and overly scattered inverted indexes cannot fully utilize the 256B space during queries, causing increased latency.

[0083] Therefore, when writing new data to the inverted index, this invention proposes a fine-grained inverted index merging method for low-frequency and overly scattered inverted records. This method consists of two parts: delayed writing and background merging. Delayed writing involves delaying the writing of the inverted table of low-frequency words in memory to the NVM, such as... Figure 7 As shown, in the process of writing the inverted index to the NVM, the length count of the inverted table for each term in memory is stored. The inverted table is filtered according to the size of count. Only when count is greater than or equal to a set size C is the inverted table saved to the NVM. Inverted tables with a length less than C are delayed in being written to the NVM. This method can greatly reduce the dispersion of the inverted index. Considering the effects of timely data persistence and delayed writing, low-frequency inverted tables will also be written to the NVM if the count still does not meet the size requirement after 3 virtual segment writes.

[0084] Delayed writes can significantly reduce the problem of an overly scattered inverted index, but to ensure the inverted index is persisted as quickly as possible, the delay should not be too long. For the remaining low-frequency inverted data, only merging can be used. This invention employs a background merging method. The NVM dictionary stores the total length L of the inverted index for each term and the number of virtual segments M. The average length p of the inverted record table stored in each virtual segment can be obtained based on the ratio of L / M. The size of p determines whether the inverted records for that term are too scattered. If p is too small, the inverted index is too scattered, and the term is marked as needing merging. When the system is idle, background merging is performed on the inverted records found during the query process, and the inverted records are reallocated to new virtual segments based on the remaining space in the virtual segments. Figure 8 The process is illustrated step-by-step: ① A background query process identifies overly scattered inverted indexes like term2 and term3 from the NVM dictionary; ② These terms are recorded separately at the end of the NVM dictionary; ③ When the system is idle, a background merging process loads the overly scattered inverted index terms from the NVM dictionary; ④ The inverted indexes, location information, etc., are loaded and merged based on the term addresses; ⑤ Virtual segments with sufficient remaining space are selected from the merged inverted indexes for storage, and the corresponding address array in the NVM dictionary is modified. When modifying data within virtual segments, PMDK's transactional read / write operations are used for data such as the length of the virtual segment array and the length within each virtual segment.

[0085] (iv) Index Migration

[0086] The inverted index structure based on NVM needs to be within NVM to achieve its full performance, but inverted indexes often need to be migrated to other non-NVM devices. Therefore, the issue of NVM index migration needs to be addressed. Simply copying the NVM inverted index structure to disks or other block devices can negatively impact query performance due to the poor randomness of these devices. To address this, this invention designs an NVM index migration method. When the inverted index in NVM becomes cold data or the virtual segments fill the entire NVM pool, the entire inverted index is merged in the background and reorganized into a commonly used inverted index storage structure. This structure is then written to disks or SSDs, facilitating the saving, migration, and querying of the inverted index from NVM. In the index migration part, the NVM dictionary structure is reorganized, and the inverted data in the virtual segments is reloaded and merged. The merged structure exhibits better query performance on disk.

[0087] In summary, this invention designs an inverted index merging method based on dictionary merging, taking into account the physical characteristics of NVM. It also designs a corresponding inverted index dictionary structure and virtual segment structure. By replacing traditional inverted index segment merging with dictionary merging, the performance overhead of creating the inverted index is reduced, improving the creation speed of the inverted index without increasing the query latency of most terms. This invention designs a fine-grained inverted index merging method, delaying the writing of low-frequency inverted tables to avoid generating a large number of overly scattered low-frequency inverted records. Furthermore, it performs fine-grained merging in the background based on the filtered overly scattered inverted records, effectively reducing the number of overly scattered inverted tables and the size of the dictionary. This invention achieves index portability, reorganizing the NVM inverted index into a currently common inverted index structure, which can achieve good performance on block devices such as disks, improving the practicality of the NVM inverted index.

[0088] Simulation Experiment

[0089] Experimental Environment Introduction: The relevant experimental environment for this invention is as follows:

[0090] CPU: Intel Xeon Gold 5218

[0091] Memory: 256GB

[0092] NVM: 1st Generation Optane Memory 1TB

[0093] SSD: 2TB

[0094] Lucene++ version: 3.0.1

[0095] GCC version: 8.4.0

[0096] IPMCTL version: 02.00.00.3709

[0097] PMDK version: 1.10

[0098] Dataset: The corpus used in the experiment was the English Wikipedia corpus EnWiki20210801, which contains over 6.3 million entries and their corresponding explanations. This paper partitioned the entire corpus based on the entries and their explanations, resulting in over 6.3 million English documents, with a text data volume of approximately 20GB. Subsequent tests, including the creation of an inverted index and querying, were then conducted using this data.

[0099] Evaluation metrics: This invention optimizes the inverted index for NVM, which speeds up the creation of the inverted index. Therefore, the main evaluation metrics are the inverted index creation time and query latency.

[0100] Experimental results:

[0101] The two most basic application scenarios for inverted indexes are index creation and query testing. First, we conduct index creation testing. Since this paper mainly focuses on the storage of the inverted index, it is unrelated to the tokenization process, and the time for tokenization using the same tokenizer is the same. Therefore, this paper pre-segments and stores the document set to avoid interference from tokenization and resulting unclear results. As a comparison, this paper implements a control group inverted index based on the latest Lucene storage structure and index merging strategy, called the Lucene structure inverted index. The inverted index designed in this paper is called the NVM inverted index. Both use the same memory structure and compression method during creation, and the entire inverted table is maintained in memory using a map structure.

[0102] Figure 10 The comparison of index creation time shows that this invention accelerates the creation of the inverted index, reducing creation time by nearly 20%. This is primarily because the dictionary occupies very little space in the inverted index, while the inverted table and location information store a large amount of data. During the inverted index merging process, a complete merge requires migrating the dictionary, inverted table, and location information structures simultaneously. Although the inverted index merging design in this paper increases the dictionary size, the added dictionary size is still far smaller than that of a complete inverted index structure. Throughout the merging process, the inverted index designed in this paper only merges the dictionary; the amount of data moved and copied is far less than that of a complete inverted index, resulting in a shorter merging time and faster inverted index creation performance.

[0103] Since the index update process is similar to the index creation process, this paper also conducted an update test of the inverted index. Based on the existing inverted index, 10,000 new documents were added, and a set of index update time comparisons were obtained, such as... Figure 11 As shown, the update process here is not the usual creation of an additional index, but rather the complete merging of the updated data into the existing inverted index. Because the amount of data merged between the two inverted indexes differs significantly, the update speeds also differ significantly; as can be seen, the NVM inverted index update time is reduced by 89%.

[0104] Queries are a frequent use case for inverted indexes, and changes to the underlying storage structure of the inverted index also have a certain impact on queries. This paper conducted index query tests, and the inverted indexes used in the queries were all stored on NVM. A comparison was made between the inverted index of the Lucene storage structure and the NVM inverted index structure proposed in this study in terms of query performance. The results are as follows: Figure 12 As shown. Due to the varying lengths of documents, to avoid performance issues caused by outputting document content, only the document ID is returned here.

[0105] Considering the inconsistent loading time of inverted indexes with different word frequencies, the experiment conducted query tests for low-frequency, mid-frequency, and high-frequency words. Low-frequency words were defined as those with fewer than 1000 results, mid-frequency words as those with around 5000 results, and high-frequency words as those with over 50,000 results. The average of 10 results for each frequency was used. In the unmerged case, the dictionary and inverted index data are stored separately, requiring queries to find offsets from multiple dictionaries and load data from multiple segments, resulting in the highest latency. For Lucene's storage structure, the entire inverted index was completely merged; this paper performed a complete dictionary merge. It can be seen that both methods return roughly the same results for high-frequency and mid-frequency word queries, but the latency for low-frequency words is significantly increased.

[0106] This invention uses dictionary merging instead of the traditional whole inverted index merging, thus reducing the amount of data written to the inverted index. This paper further tests the amount of data written during index creation. The experiment chose to build the complete inverted index without saving the original document text, so the forward index information occupies little space. To better simulate the Lucene storage structure's index creation process, this paper also uses Lucene to create an index on the dataset. During the index creation process, two rounds of index segment merging and two rounds of intra-segment merging occurred, resulting in the same data being repeatedly written four times. The experimentally implemented Lucene structure implemented two rounds of index segment merging during the index creation process. Considering that intra-segment merging could be performed without it, this experiment did not implement intra-segment merging. The data written volume during index creation for the unmerged, Lucene structure, and this invention is as follows: Figure 13 As shown, an unmerged index only requires writing index data once, resulting in a smaller overall size; Lucene stores data by merging the entire inverted index twice, leading to a larger amount of data written; compared to the Lucene structure, the NVM inverted index only merges the dictionary, reducing the amount of data written by nearly 57% compared to the Lucene structure.

[0107] To address the latency of low-frequency words, this invention designs a fine-grained inverted index merging method, which reduces the query latency of low-frequency words and also reduces the size of the dictionary. Figure 14 This refers to the change in dictionary size after merging fine-grained inverted indexes. Figure 15 This shows the change in query latency for low-frequency words. Here we compare the query latency of 10 low-frequency words.

[0108] Since the inverted index designed in this paper is mainly used on NVM, its performance will significantly decrease after the storage medium is replaced. Figure 16This paper compares the query latency of high-frequency terms for two inverted index structures on NVM and SSD. The query latency of 10 high-frequency terms was selected and averaged. It can be observed that placing the NVM inverted index designed in this paper on SSD significantly increases query latency. Considering the capacity of NVM and the compatibility of current NVM hardware, index portability is necessary. Therefore, during the migration process, a complete inverted index merging of the NVM inverted index is required. Experiments demonstrate that this invention improves inverted index creation performance while refining the inverted index storage structure.

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. An inverted index storage device based on non-volatile memory, comprising: At least one virtual segment is used to store basic information about the inverted index; An NVM dictionary is used to maintain the basic information in each virtual segment based on the virtual segment dynamic address array contained in each term in the NVM dictionary; The term and inverted index basic information are obtained by segmenting the document into words and building an inverted index in memory. The inverted index basic information includes: inverted record table, word position information and skip list; The structure of the virtual segment includes: a segment header, a data segment, and a segment footer, wherein, The data segment is used to store basic information about the inverted index; The segment tail includes: redundant space and segment tail identifier; The segment header includes: segment identifier, inverted index, data length, address range, and redundancy space size, wherein, The inverted index is used to record the number of terms corresponding to the basic information of the inverted index in the current virtual segment; The data length is used to record the number of bytes in the current virtual segment; The address range is used to record the start and end addresses of the current virtual segment in the cache pool of the non-volatile memory; The redundancy space size is used to record the size of the redundancy space in the current virtual segment; The structure of the NVM dictionary includes: dictionary version, number of terms, skip list span, and term content, wherein the term content includes: PreLen is used to record the number of times the prefix of the term is the same as that of the previous term; Suffix is ​​used to indicate a word suffix; FieldNum is used to represent the number of fields in which the term appears; Count indicates the total number of times the term appears; The DocFreq array represents the total number of documents in which this term appears; The Skips array is used to record the address and document number of the corresponding skip list in each virtual segment, wherein the skip list is stored in the data segment.

2. A method for creating and incrementally updating an inverted index in an inverted index storage device as described in claim 1, comprising the following steps: After word segmentation of the document, an inverted index structure is built in memory. The inverted index structure includes basic information about terms and inverted indexes. If the space occupied by the inverted index structure in memory reaches the set first threshold, a virtual segment is created in the non-volatile memory cache pool file, and the basic information of the inverted index is stored in the virtual segment; A single-segment dictionary corresponding to a single virtual segment is created in memory, and the single-segment dictionary is saved to a non-volatile memory cache pool. The dictionary structure of the single-segment dictionary is the dictionary structure commonly used in traditional inverted indexes. The document set is continuously processed, and new virtual segments are continuously generated in the non-volatile memory cache pool file. The corresponding single-segment dictionary is appended to the non-volatile memory cache pool. When the number of single-segment dictionaries reaches the second threshold, the existing single-segment dictionaries are merged, and the merged dictionary structure is the NVM dictionary structure. When document set processing is complete, all dictionaries in non-volatile memory are merged into a single NVM dictionary, which contains all terms in the entire document set. At this point, inverted index creation or incremental updates are finished.

3. The method as described in claim 2, characterized in that, During the inverted index merging process, only the dictionary is merged. A virtual segment is created in the non-volatile memory cache pool file to store the basic information of the inverted index. It also includes: determining whether the basic information of the inverted index of the term is written to the non-volatile memory based on the length of the inverted table in memory. Compare the length with a third threshold: If the length is not less than the third threshold, then the basic information of the inverted index is stored in non-volatile memory; If the length is less than the third threshold, the basic information of the inverted index is delayed until the accumulated length of the inverted record table is not less than the third threshold or the number of delayed writes reaches the set value, and then the basic information of the inverted index is stored.

4. A background merging method for low-frequency and scattered inverted index information in the inverted index storage device as described in claim 1, comprising the following steps: Find the total length of the inverted index and the number of virtual segments containing the inverted index for each term in the NVM dictionary; Based on the total length of the inverted index and the number of virtual segments, calculate the average length p of the inverted index stored in each virtual segment; Compare the length p of the inverted index table with the fourth threshold: If the average inverted index length p is not less than the fourth threshold, then the inverted index of that term will not be processed. If the average inverted index length p is less than the fourth threshold, the inverted index of the term is merged, and a virtual segment is selected or created for storage.

5. A storage medium storing a computer program, wherein, The computer program is configured to execute any of the methods described in claims 2-4 at runtime.

6. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform the method as claimed in any one of claims 2-4.

Images