Inverted index iteration self-index optimization method based on top-k query model

Abstract

The application discloses an inverted index iterative self-index optimization method based on a Top-K query model, first, an initial inverted index is constructed with the aid of an open source tool, which is converted into an equal word compression format and reorganized into a block interleaved storage structure; subsequently, based on the Top-K query model, a multi-layer iterative self-index is constructed through recursive segmentation and sampling; further, a query performance model containing parameters such as disk I / O and decompression speed is established, the relationship between query benefits and storage overhead is quantified, and the optimal synchronous distance and the number of iterative layers are solved; finally, fast random access of a mass disk compressed inverted index is realized. Through the variable-length synchronous distance design, the Top-K model parameter optimization and the multi-layer iterative index structure, the application significantly improves the query efficiency, reduces the data reading and decompression overhead, can maintain the Pareto optimal balance of the query performance and the storage efficiency in different hardware environments, and is suitable for a mass data high-concurrency query scene.

Description

Technical Field

[0001] This invention belongs to the field of inverted index query technology for search engines, specifically involving an inverted index iterative self-indexing optimization method based on the Top-K query model. Background Technology

[0002] Inverted indexes, as the core data structure for search engines to achieve efficient text retrieval, directly determine the response speed, throughput, and service quality of search engines. To cope with the storage pressure of massive amounts of data and improve query efficiency, inverted index compression technology has become an indispensable key component in this field. Among numerous compression schemes, word-aligned compression algorithms have become a research hotspot for both academia and industry due to their ability to achieve a good balance between compression ratio, decompression speed, and hardware friendliness. These algorithms can be mainly divided into two schools: equal-length compression algorithms and equal-word compression algorithms. The former, such as PFOR and VSE, compress data in fixed-length segments. While offering high decompression efficiency, they may experience a decrease in compression ratio due to outliers within segments and may face bottlenecks in cross-block decompression during disk reads. The latter, represented by Simple-9, Simple-16, and Simple-8b, uses a set of predefined, fine-grained compression patterns within a fixed-size machine word (e.g., 32-bit or 64-bit) to pack as many integer sequences as possible.

[0003] Equal-word compression algorithms, through their ingenious pattern design, ensure that the compressed codewords are strictly aligned with machine word boundaries. This characteristic not only brings high compression density and fast decompression performance, but more importantly, it fundamentally avoids the "cross-block decompression" problem commonly encountered when reading compressed data from disk. That is, it eliminates the need for additional I / O operations caused by a compressed codeword spanning two physical disk blocks. This advantage makes it particularly suitable for scenarios with massive disk-resident inverted indexes. However, due to its mechanism of packing integer sequences into fixed-length blocks, it naturally produces compressed data segments of varying lengths, directly leading to the unpredictability of self-index synchronization distance. Traditional self-indexing methods based on fixed intervals fail in this scenario because the preset synchronization point may fall inside a compressed codeword, making it impossible for the system to directly calculate the target address offset without decompressing the preceding data, thus severely weakening the fast random access capability that the index structure should provide.

[0004] Existing self-indexing structures are primarily designed for full traversal of inverted lists or single-point queries, failing to adequately align with the dominant Top-K query pattern in search engines. In this pattern, dynamic pruning algorithms such as WAND and MaxScore require only accessing and evaluating a few high-potential inverted items during the query process, exhibiting a significant "partial access" characteristic. The static design of traditional self-indexes in terms of synchronization distance and hierarchical structure cannot intelligently adapt to this access pattern, easily leading to wasted storage space or insufficient query skipping efficiency, making it difficult to achieve a fine-grained optimal balance between query acceleration benefits and additional storage costs.

[0005] Furthermore, a collaboration gap exists between current self-indexing and Top-K query execution engines. Self-indexing is typically treated as an independent positioning module, failing to be deeply integrated into the decision-making logic of the query process (such as access control based on term weights and upper bounds of document scores). This prevents the system from intelligently determining the necessity and scope of exploration within multi-level indexes, potentially leading to unnecessary index layer decompression and data block loading. Simultaneously, the "greedy" strategy employed by equal-word compression algorithms when handling interleaved data streams wastes compression bit width, and its frequent mode selection and branching decisions introduce considerable CPU overhead. These inherent limitations collectively restrict its ultimate performance potential in scenarios involving massive data and high-concurrency queries. Summary of the Invention

[0006] The purpose of this invention is to provide an inverted index iterative self-indexing optimization method based on the Top-K query model, which improves the random access efficiency and overall query performance of massive disk inverted indexes under real query loads.

[0007] The technical solution adopted in this invention is an inverted index iterative self-index optimization method based on the Top-K query model, which is implemented according to the following steps: Step 1: By generating a dictionary with compression adaptation attributes, construct an integer sequence type inverted linked list and output the initial inverted index for adapting equal-character compression; Step 2: Convert the initial inverted index built in Step 1 into an equal-word compressed format adapted to the iterative self-index construction; Step 3: Using the inverted index in equal-word compressed format output in Step 2 as input, adapt the compressed data characteristics, storage structure and preliminary segmentation results of Step 2 to establish a Top-K query performance model for the self-index, determine the optimal synchronization distance and iteration layer, and finally form a self-index structure adapted to massive data.

[0008] The invention is further characterized in that, Step 1 specifically involves: Step 1.1: Traverse all documents, extract all words contained in each document and summarize them into the same list, perform deduplication on the list to form a dictionary of words without duplicates; Step 1.2: Read the word dictionary list file and the document collection list file with assigned document numbers; Step 1.3: Create an empty word inverted list to store the complete adapted compressed inverted index data. This word inverted list is used to store the entire inverted index. Step 1.4, Outer Loop: Retrieve the current word, initialize a temporary dictionary to store the inverted linked list information of the word, set the 'word' key in the temporary dictionary with the value of the current word, set the 'index' key with the value of an empty list to store the document number containing the word; Step 1.5, Inner Loop: Retrieve the current document information, determine whether the word segmentation list of the current document contains the word of the outer loop, if it does, add the document's index to the 'index' list of the temporary dictionary; if it does not, continue processing the next document. Step 1.6: After completing the inner loop, add the temporary dictionary of the current word to the inverted word list created in Step 1.3; Step 1.7, repeat steps 1.4-1.6 until all entries in the word dictionary have been processed. The final output initial inverted index is the word dictionary and the inverted linked list of words.

[0009] Step 2 specifically involves: Step 2.1: Read the initial inverted index generated in Step 1, extract the inverted linked list corresponding to each term; perform preprocessing on each linked list, verify and ensure that the document number is strictly incremental, convert the document number into a difference sequence, and separate the term frequency sequence, so that each linked list is finally converted into two independent integer sequences: difference and term frequency. Step 2.2: Apply equal-word compression to compress the difference and word frequency integer sequences obtained in Step 2.1 independently. The compression is performed in 32-bit / 64-bit machine words, continuously encoding integers until the available data bits of the current machine word are filled, generating variable-length compressed data segments, and recording the starting address offset of each compressed data segment. Step 2.3 processes two sets of compressed data from the same inverted index: one set is compressed data of document number differences, and the other set is compressed data of the frequency of words in the corresponding documents, resulting in an inverted index in equal-character compressed format; these two sets of data are then stored alternately together, forming a physical storage block, according to their original order corresponding to each document. Meanwhile, an initial lower limit d for synchronization distance is set, and the entire linked list is logically divided: starting from the beginning of the linked list, the data is decompressed one by one and the number of inverted items is counted. When the cumulative number of inverted items exceeds d for the first time, a mark is made at this position and it is regarded as a logical segmentation boundary. After all storage and segmentation operations are completed, the final inverted index in equal-word compressed format is obtained.

[0010] Step 3 specifically involves: Step 3.1: Establish a Top-K query performance model with a single-layer self-index to obtain the optimal number of synchronization points; Step 3.2: Based on the Top-K query performance model of a single-layer self-index, add a second-layer self-index structure, derive the two-layer overhead model, and determine the optimal number of synchronization points for the second layer; Step 3.3, Calculation The Top-K query performance model of the layer-iterative self-index structure is used to obtain the optimal synchronization distance and iteration layer number of the self-index structure of each inverted linked list, and finally form a self-index structure adapted to massive data.

[0011] Step 3.1 specifically involves: Let the length of any inverted list in the inverted index obtained after step 2 be . Suppose a single-level self-index structure contains There are 1 optimal synchronization points; let the synchronization distance be . ; Total query time overhead Includes disk read overhead Data decompression overhead and data search overhead ,Right now

[0012] The time taken to read each inverted item from disk is , The physical storage block size is determined in step 2.3; the read overhead includes the overhead of reading the self-indexed structure generated in step 2 and the physical storage block reassembled in step 2, i.e.

[0013] Let the decompression time for each inverted index be... The decompression overhead includes the overhead of the self-indexed structure generated in decompression step 2 and the overhead of the compressed data segments within the logical segments marked in decompression step 2, i.e.

[0014] The average lookup cost for a single data segment is the same as that for a single-level self-indexed structure. The search for each synchronization point and the search within the physical storage block reassembled in step 2, i.e.

[0015] Therefore, the worst-case search performance model is derived as follows:

[0016] By finding the minimum point through differentiation, the worst-case search performance model formula is obtained. smallest The value is ; Based on the performance of mainstream servers, the optimal number of synchronization points for a single-level self-indexing structure is inferred. The distribution of synchronization points must be aligned with the boundaries of the compressed data segments generated in step 2. Synchronization points are set according to the data segment length: if the length of the compressed data segment generated in step 2 is less than the lower limit of the synchronization distance... Set the inverted item corresponding to the right boundary of this compressed data segment as the synchronization point; if the length of the compressed data segment generated in step 2 is less than the lower limit of the synchronization distance. This short data segment is then merged sequentially with one or more subsequent compressed data segments until the total number of inverted items in the merged whole exceeds the lower limit of the synchronization distance. Then, set the inverted item corresponding to the right boundary of the merged whole as the synchronization point.

[0017] Step 3.2 specifically involves: Suppose the second-level self-index structure contains Based on the derivation process in step 3.1, the worst-case query performance model under this self-index structure can be derived as follows:

[0018] Solving the above formula makes To obtain the minimum value ( , The combination of these two layers of self-indexing is used to configure the optimal synchronization point for the two layers, and the synchronization distance of the second layer can be calculated.

[0019] Step 3.3 specifically involves: calculate A Top-K query performance model using a layered iterative self-indexing structure, wherein... To determine the length of the inverted linked list from step 2 and physical storage block size Given a definite theoretical maximum number of layers, let... ;No. The optimal number of synchronization points for a layered self-indexed structure is ; Using the above derivation process, the worst-case query performance model under this self-index structure can be obtained as follows:

[0020] From the above formula, we can derive that... Minimum of all Values ​​and synchronization distances; by comparing the results under different self-indexing iteration levels. The value determines the optimal number of iteration layers, when When the optimal iteration level of the query performance model is determined, the optimal synchronization distance and iteration level of the self-index structure of each inverted linked list can be obtained, ultimately forming a self-index structure adapted to massive amounts of data.

[0021] The beneficial effects of this invention are: (1) To address the variable length characteristics of equal-word compressed inverted indexes, a self-index structure with variable-length synchronization distance is proposed to improve random access efficiency; (2) Combining the Top-K query model to optimize the self-index parameters significantly reduces the data reading and decompression overhead during the query process; (3) By constructing a multi-level iterative self-index, the data segment location is further accelerated, adapting to the massive disk indexing environment; (4) Provide a complete query performance model and parameter optimization method to ensure that the self-index structure can maintain optimal performance in different hardware environments. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the inverted index iterative self-index optimization method based on the Top-K query model of the present invention; Figure 2 This is a logical view and physical storage diagram of an iterative self-indexed structure; Figure 3 Example diagram of a storage format for word compression. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0024] Example 1 This invention relates to an inverted index iterative self-index optimization method based on the Top-K query model, such as... Figure 1 As shown, please follow these steps: Step 1: Construct the initial inverted index for adaptive character compression: By generating a dictionary with compression adaptation attributes, construct an integer sequence type inverted linked list and output the initial inverted index for adaptive character compression; specifically: Step 1.1: Traverse all documents, extract all words contained in each document and summarize them into the same list, perform deduplication on the list to form a dictionary of words without duplicates; Step 1.2: Read the word dictionary list file and the document collection list file with assigned document numbers; Step 1.3: Create an empty word inverted list to store the complete adapted compressed inverted index data. This word inverted list is used to store the entire inverted index. Step 1.4, Outer Loop: Retrieve the current word, initialize a temporary dictionary to store the inverted linked list information of the word, set the 'word' key in the temporary dictionary with the value of the current word, set the 'index' key with the value of an empty list to store the document number containing the word; Step 1.5, Inner Loop: Retrieve the current document information, determine whether the word segmentation list of the current document contains the word of the outer loop, if it does, add the document's index to the 'index' list of the temporary dictionary; if it does not, continue processing the next document. Step 1.6: After completing the inner loop, add the temporary dictionary of the current word to the inverted word list created in Step 1.3; Step 1.7, repeat steps 1.4-1.6 until all entries in the word dictionary have been processed. The final output initial inverted index is the word dictionary and the inverted linked list of words.

[0025] Step 2: Convert the initial inverted index built in Step 1 into an equal-word compressed format adapted for iterative self-index construction; for example... Figure 3 As shown, specifically: Step 2.1, Data Extraction and Preprocessing: Read the initial inverted index generated in Step 1 and extract the inverted linked list corresponding to each term; perform preprocessing on each linked list: verify and ensure that the document number is strictly incremental, convert the document number into a difference sequence, and separate the term frequency sequence, so that each linked list is finally converted into two independent integer sequences: difference and term frequency.

[0026] Step 2.2: Apply equal-word compression, using Simple-9 / Simple-16 / Simple-8b and other equal-word compression algorithms to independently compress the difference and word frequency integer sequences obtained in Step 2.1. The compression is performed in 32-bit / 64-bit machine words, continuously encoding integers until the available data bits of the current machine word are filled, generating variable-length compressed data segments, and recording the starting address offset of each compressed data segment. Step 2.3 processes the two sets of compressed data from the same inverted index: one set is compressed data of document number differences, and the other set is compressed data of the frequency of words in their corresponding documents. Following their original order corresponding to each document, these two sets of data are stored alternately to form actual storage units (physical storage blocks). Simultaneously, an initial lower limit of synchronization distance d is set (e.g., d=64), and the entire linked list is initially logically divided: starting from the beginning of the linked list, the data is decompressed one by one and the number of inverted items is counted. When the cumulative number of inverted items exceeds d for the first time, a mark is made at this position, which is regarded as the logical segment boundary. After all storage and segmentation operations are completed, the final inverted index in equal-word compressed format is obtained; Step 3: Using the inverted index in equal-word compressed format output from Step 2 as input, adapt the compressed data characteristics, storage structure, and preliminary segmentation results from Step 2 to establish a Top-K query performance model for the self-index, determine the optimal synchronization distance r and the number of iteration layers h, and finally form a self-index structure adapted to massive amounts of data; specifically: Step 3.1: Establish a Top-K query performance model with a single-layer self-index to obtain the optimal number of synchronization points. Specifically: Let the length of any inverted list in the inverted index obtained after step 2 be . Suppose a single-level self-index structure contains Find the optimal synchronization point. Since step 2 uses an equal-word compression algorithm (such as Simple-9 / Simple-16 / Simple-8b) to generate variable-length compressed data segments, let the variable-length synchronization distance be limited to... (r is also the total number of inverted items between two adjacent synchronization points in a single-level self-index; the Simple-9 equal-word compression algorithm selected in step 2 can compress a maximum of 28 integers in a single pass, and the synchronization distance is...) .

[0027] Total query time overhead Includes disk read overhead Data decompression overhead and data search overhead ,Right now

[0028] The time taken to read each inverted item from disk is , This refers to the physical storage block size determined in step 2.3. The read overhead includes the overhead of reading the self-indexed structure generated in step 2 and the physical storage blocks (inverted linked list data blocks) reorganized in step 2, i.e.

[0029] Let the decompression time for each inverted index be... The decompression overhead includes the overhead of the self-indexed structure generated in decompression step 2 and the overhead of the compressed data segments within the logical segments marked in decompression step 2, i.e.

[0030] The average lookup cost for a single data segment is the same as that for a single-level self-indexed structure. The search for each synchronization point and the search within the physical storage block reassembled in step 2, i.e.

[0031] Therefore, the worst-case search performance model is derived as follows:

[0032] By finding the minimum point through differentiation, the worst-case search performance model formula is obtained. smallest The value is .

[0033] Based on the current mainstream server performance (obtaining parameters such as disk I / O speed and data decompression speed), the optimal number of synchronization points for a single-level self-index structure is inferred. The distribution of synchronization points must be aligned with the boundaries of the compressed data segments generated in step 2 (to avoid subsequent queries where synchronization points fall within a single compressed data segment, requiring additional decompression of preceding data to locate the target, thus losing the advantage of fast access). Synchronization points are set according to the data segment length: if the length of the compressed data segment generated in step 2 is greater than the lower limit of the synchronization distance... Set the inverted item corresponding to the right boundary of this compressed data segment as the synchronization point; if the length of the compressed data segment generated in step 2 is less than the lower limit of the synchronization distance. This short data segment is then merged sequentially with one or more subsequent compressed data segments until the total number of inverted items in the merged whole exceeds the lower limit of the synchronization distance. Then, set the inverted item corresponding to the right boundary of the merged whole as the synchronization point.

[0034] Step 3.2: Based on the Top-K query performance model of a single-layer self-index, add a second-layer self-index structure, derive the two-layer overhead model, and determine the optimal number of synchronization points for the second layer. ; Suppose the second-level self-index structure contains Based on the derivation process in step 3.1, the worst-case query performance model under this self-index structure can be derived as follows:

[0035] Solving the above formula makes To obtain the minimum value ( , The combination of these elements is ultimately configured as the optimal synchronization point for the two-layer self-indexed system, and the synchronization distance of the second layer can be calculated. Step 3.3, Calculation A Top-K query performance model using a layered iterative self-indexing structure, wherein... To determine the length of the inverted linked list from step 2 and physical storage block size Given a definite theoretical maximum number of layers, let... ;No. The optimal number of synchronization points for a layered self-indexed structure is ; Using the above derivation process, the worst-case query performance model under this self-index structure can be obtained as follows:

[0036] From the above formula, we can derive that... Minimum of all Values ​​and synchronization distance. This is determined by comparing values ​​under different self-indexing iteration levels. The value determines the optimal number of iteration layers, when When the optimal iteration level of the query performance model is determined, the optimal synchronization distance and iteration level of the self-index structure of each inverted linked list can be obtained, ultimately forming a self-index structure adapted to massive amounts of data.

[0037] Example 2 This invention presents an inverted index iterative self-indexing optimization method based on the Top-K query model. By accurately quantifying the trade-off between query performance and storage overhead, and adaptively optimizing the synchronization distance and the number of iteration layers, it significantly improves the random access efficiency and overall query performance of massive disk inverted indexes under real query loads while ensuring low storage overhead.

[0038] Example 3 Basic data preparation: 1. Document Collection: 50 documents in total, numbered... As shown in Table 1, the documents containing the term "algorithm" and their frequencies; Table 1. Documents and word frequencies

[0039] 2. Initial Inverted Index: The inverted list for the term "algorithm" is index=[2,5,7,9,12,15,18,21,24,27,30,33,36,39,42,45,48,50,…]. The final inverted list has a length p=120 and contains 120 document numbers.

[0040] Step 1: Construct the initial inverted index for adapting to equal-word compression. Follow steps 1.1-1.7 to generate a temporary dictionary for the term "algorithm": {"word":"algorithm","index":[2,5,7,9,12,...,48,50]} Step 2: Convert to an equal-word compression format that supports iterative self-indexing. Step 2.1, Data Preprocessing Generate the d-gap difference sequence: Calculate the difference between adjacent document numbers, and finally obtain a sequence of 120 differences: d-gap=[2,3,2,2,3,3,3,3,3,3,…,2].

[0041] Separate the word frequency sequence: Generate a sequence of 120 word frequencies, freq=[1,2,1,3,2,1,2,1,3,2,…,1], corresponding one-to-one with the difference sequence.

[0042] Step 2.2, apply Simple-9 word compression, etc. Simple-9 supports multiple modes for packing integers. Selecting mode 5 (which can pack 10 integers) compresses them in 32-bit machine words. Difference sequence compression: Each data segment consists of 10 differences, resulting in a total of 120 ÷ 10 = 12 compressed data segments.

[0043] Frequency sequence compression: Corresponding one-to-one with the difference sequence, it is also divided into 12 compressed data segments. The starting address offset of each data segment is recorded (e.g., 0x00, 0x04, 0x08, ..., 0x2C).

[0044] Step 2.3, Reorganize Storage and Preliminary Logical Segmentation. Interleaved Storage: Interleave the difference segments and word frequency segments sequentially to form physical storage blocks; Storage order: Difference segment 1 → Word frequency segment 1 → Difference segment 2 → Word frequency segment 2 → ... → Difference segment 12 → Word frequency segment 12; Preliminary segmentation: Set the lower limit of synchronization distance d=64 The first logical segment contains a total of 64 inverted items (covering the first 6 difference segments + 6 word frequency segments). The second logical segment contains 56 remaining inverted items. Since 56 < 64, they are merged into one logical segment.

[0045] Step 3: Construct a single-layer self-indexed Top-K optimization model Step 3.1, define the core parameters: inverted list length p = 120; disk read time. Inverted items; decompression time Inverted index; physical block size b=64; Simple-9 can compress a maximum of 28 integers at a time, synchronization distance .

[0046] Calculate disk read overhead :

[0047] Calculate the overhead of data decompression :

[0048] Calculate data lookup cost :

[0049] Solving for the optimal number of synchronization points :

[0050] Synchronization distance and data segment adjustment: Lower limit of synchronization distance .

[0051] Data segment adjustment: For compressed data segments longer than 35.57, the synchronization point is set to the right boundary; for segments shorter than 35.57, a synchronization point is set after merging. This results in 7 synchronization points, with synchronization distances ranging from 36 to 40. Calculate the total query cost of a single layer Substitution : can be obtained ; ; The final total cost of a single layer can be obtained. =7.1+2.06+21.29≈30.45μs.

[0052] Step 3.2, Construct a two-layer self-indexing optimization model Substitution , The formula for the total cost of a two-level system is:

[0053] Find the optimal number of synchronization points in the second layer. : Through numerical solution, when hour, To obtain the minimum value: Calculated Total cost of two-level This is approximately 27.1% lower than that of a single-layer structure.

[0054] Step 3.3: Construct an h-layer iterative self-indexed globally optimal model and calculate the total cost for different layer numbers: (Calculate the total cost for h=1~5 sequentially) Number of iterations h=1, total cost =30.45 μs The number of iterations, h=2, and the total cost is... =22.18 μs The number of iterations, h=3, and the total cost is... =18.62 μs The number of iterations, h=4, and the total cost is... =18.25 μs The number of iterations, h=5, and the total cost is... =18.91 μ s ; Determine the optimal number of iterations: When h=4, the total cost is... The minimum value is approximately 18.25 μs, at which point the number of synchronization points in each layer is... , , 1. .

[0055] The general formula for performance improvement rate is:

[0056] 4-layer iterative self-indexing vs. no self-indexing (57.2% improvement): Time before optimization (no self-indexing): 42.6μs; Time after optimization (4-layer iterative self-indexing): 18.25μs; Substitute into the formula to calculate:

[0057] Two-level iterative self-indexing vs. no self-indexing (48% improvement): Time before optimization (no self-indexing): 42.6μs; Time after optimization (two-level iterative self-indexing): 22.18μs; Substitute into the formula to calculate:

[0058] Single-layer iterative self-indexing vs. no self-indexing (40.0% improvement): Time before optimization (no self-indexing): 42.6μs; Time after optimization (single-layer iterative self-indexing): 30.45μs; Substitute into the formula to calculate:

[0059] The experimental results are shown in Table 2: Table 2 Comparison of response times between iterative self-indexed and non-self-indexed systems.

[0060] Example 4 Hardware configuration: Intel Xeon Quad-Core processor, 8GB RAM, SATA hard drive.

[0061] Software platform: Built on Python 3.8 and the open-source search engine Terrier 4.1.

[0062] Step 1: Construct the initial inverted index for adapting to equal-word compression. This step uses the TREC WT2G standard dataset as the original input to generate an initial inverted index adapted for equal-word compression, providing basic data for subsequent experiments.

[0063] Step 1.1: Generate a word dictionary Preprocessing was performed on the web documents in the TREC WT2G dataset: HTML tags were removed, and the titles, body text, and anchor text were extracted. The Porter stemming analyzer was used to remove English stop words and perform stemming. The word segmentation results of all documents were iterated and compiled into a word dictionary containing 89,276 unique terms, which was then stored as a text file.

[0064] Step 1.2: Prepare the input data by assigning unique and continuously increasing integer serial numbers to the 32041 documents, and organizing them into a JSON file of "document serial number - document word segmentation list". Simultaneously, load the word dictionary list generated in step 1.1 as the base data for subsequent loop iterations.

[0065] Step 1.3: Initialize the storage container and create an empty list word_inverted_list to store the complete inverted index data. The list elements are a dictionary structure of {"word": term,"index": list of document numbers}.

[0066] Step 1.4, Outer Loop: Initialize the temporary dictionary of terms. Read the terms in the word dictionary in order, and initialize a temporary dictionary temp_dict for each term, containing two keys: "word" and "index".

[0067] Step 1.5, Inner Loop: Match documents and record their serial numbers. Read the "document serial number - document word segmentation list" one by one and determine whether the current document contains the target word. If it does, add the document serial number to temp_dict["index"]; if it does not, skip the document.

[0068] Step 1.6: Store in the inverted list container. After matching all documents for the current term, add temp_dict to word_inverted_list, ensuring that the "index" list is sorted in ascending order of document number.

[0069] Step 1.7: Loop completion and index output. Repeat steps 1.4-1.6 until all 89276 terms have been processed. The final output is the initial inverted index: a word dictionary text file and an inverted list JSON file, where the average document frequency of the inverted list is 42.3, and the longest inverted list contains 28745 document indices.

[0070] Step 2: Convert to an equal-word compression format that supports iterative self-indexing. This step converts the initial inverted index into an equal-word compressed format, selects the optimal compression algorithm based on simulation results, and generates compressed data adapted to the iterative self-index.

[0071] Step 2.1, Data Extraction and Preprocessing: Read the inverted list JSON file output from Step 1, extract the "index" list for each term, and perform the following preprocessing: Verify document number order: Ensure that all document numbers in the list are strictly in ascending order and there are no out-of-order issues; Convert the difference sequence: Convert the list of document numbers into a d-gap difference sequence. For example, the list of document numbers [3,7,12,25,...] for "computer" is converted into the difference sequence [3,4,5,13,...]. Separate the word frequency sequence: Count the number of times "computer" appears in each document, generating a word frequency sequence [2,1,3,2,...] with the same length as the difference sequence. Finally, each list is transformed into two independent integer sequences: the difference and the word frequency.

[0072] Step 2.2, apply equal-word compression. Simple-9 is chosen as the equal-word compression algorithm, and its performance advantages are as follows: Compression ratio: In the long array test of clustered data, Simple-9's compression ratio is 4.0 b / i, which is on par with Simple-9 and better than Varint and Group Varint; Compression speed: Simple-9's compression speed reaches 158.7 mi / s, which is 13.9% higher than Simple-9; Decompression speed: Simple-9's decompression speed reaches 711.3 mi / s, which is 61.3% higher than Simple-9 and significantly better than Varint.

[0073] Based on the above advantages, Simple-9 compression is implemented using 32-bit machine words: The status bit is designed as an 8-bit status bit; data storage: the 24-bit data bits of the first 32-bit machine word and the 4-bit data bits of the second 32-bit machine word are combined to store the integer of the first Simple-9 compression mode, and the remaining 28 bits store the integer of the second compression mode; compression execution: the difference sequence and the word frequency sequence are compressed independently to generate variable-length compressed data segments, and the starting address offset of each data segment is recorded.

[0074] Step 2.3: Reorganize the storage format and perform preliminary logical segmentation. Interleaved storage: Interleave the compressed codewords of the difference sequence and the compressed codewords of the term frequency sequence according to their correspondence to form physical storage blocks; Preliminary logical segmentation: Set the initial synchronization distance lower limit d=64, and count the number of inverted items by sequential decompression. When the cumulative number of inverted items first exceeds 64, it is marked as a logical segment boundary. For example, the inverted list of "computer" contains 1568 inverted items, and finally 24 logical segment boundaries are marked. This step ultimately outputs an inverted index in equal-word compressed format: a compressed data file (approximately 128MB in size) and an address mapping table (recording the data segment start offset and logical segment boundaries).

[0075] Step 3: Construct a single-layer self-indexed Top-K basic optimization model; Based on the server parameters of the simulation experiment (disk I / O speed of 250MB / s, time to decompress one inverted item of 0.05μs), taking the inverted list of "computer" (length p=1568) as an example, a single-level self-indexing optimization model is constructed.

[0076] Step 3.1, define model parameters. The preset number of synchronization points for a single-layer self-indexed system is p1, with a variable-length synchronization distance limit. In the worst case (Because Simple-9 can only compress a maximum of 28 integers at a time). Total query overhead. =Disk read overhead +Decompression overhead + Search overhead .

[0077] Step 3.2, calculate disk read overhead

[0078] Known Inverted index, disk physical data block size b=64, calculation formula is:

[0079] Step 3.3, calculate the data decompression cost.

[0080] Known The inverted index is calculated using the following formula:

[0081] Step 3.4, Calculate data lookup cost

[0082] The calculation formula is:

[0083] Step 3.5, Solve for the optimal number of synchronization points.

[0084] Will Substituting into the total cost formula and simplifying, we get the solution. That is, the optimal number of synchronization points for a single-level self-index is 22.

[0085] Step 3.6, Determine synchronization distance and adjust data segment. For reference Figure 2 The above shows the results of dividing the data segment length distribution using the inverted index with equal characters (the compression efficiency is optimal when the data segment length is between 32 and 64). Adjust the data segments of the inverted list for "computer": If the length of the compressed data segment is greater than the lower limit of r (1568 / (2×22)≈35.6), the synchronization point is set as the right boundary of the data segment; if the length of the data segment is less than 35.6, consecutive data segments are merged until the length exceeds 35.6. Finally, 22 synchronization points are formed, with synchronization distances distributed between 36 and 62.

[0086] Step 4: Construct a two-layer self-indexing model to optimize search efficiency. By adding a second self-index on top of the single-level self-index, and combining this with Table 3 (where the query cost of a two-level self-index is lower than that of a single-level index), the search efficiency can be further optimized.

[0087] Table 3 Simulation evaluation results of the self-indexed structure based on equal-word compression

[0088] Step 4.1, establish a two-level query performance model, assuming the number of synchronization points in the second-level self-index is... Construct a two-layer query performance model:

[0089] Substitution ,get about The function expression.

[0090] Step 4.2: Solve for the optimal number of synchronization points in the two layers. This is obtained through numerical calculation. hour Minimum ( ), compared to a single-level self-index The cost was reduced by 26.1%, which is consistent with the simulation results in Table 3 that "the cost of two-level self-indexed queries is lower than that of single-level queries" (the total cost in Table 3 is 0.557 when h=2, which is lower than 0.585 when h=1).

[0091] Step 5: Construct the h-layer iterative self-indexed globally optimal model The model is extended to layer h, and the optimal number of iteration layers is determined by combining simulation experiments to achieve optimal global performance.

[0092] Step 5.1: Determine the upper limit of the number of iterations. Construct a query performance model for layer h:

[0093] Step 5.2, Solve for the optimal number of synchronization points at each layer. Calculate the values ​​for h=1 to 5 sequentially. h=1: h=2: h=3: h=4: h=5: .

[0094] Combination Figure 1 Simulation results (decompression speed advantage of grouped word compression), when h=4 At the minimum, the number of optimal synchronization points in each layer is p1=22, p2=5, p3=2, and p4=1.

[0095] Step 5.3: Determine the optimal synchronization distance and number of iteration layers. Construct a 4-layer iterative self-index based on the inverted list with the parameter "computer". The synchronization distance is dynamically distributed between 36 and 62. The total storage overhead of the self-index is approximately 8.3% of that of the compressed index.

[0096] Step 5.4 outputs the final optimization results. Using the same method, the optimal self-index parameters—synchronization distance, iteration level, and synchronization point address—are calculated for the inverted index of all 89,276 terms and stored as an index metadata file. This is used in conjunction with the equal-word compressed index generated in Step 2 to quickly locate the target data segment in Top-K queries.

[0097] Example 5 Experimental verification results Based on the optimization method in Example 4, performance tests were conducted on the TREC WT2G dataset, and the results are as follows: 1) Compression performance: The compression ratio of the grouped Simple-9 reaches 4.0b / i, the compression speed is 158.7mi / s, and the decompression speed is 711.3mi / s, which is a significant improvement over the traditional Simple-9; 2) Query performance: The average response time for Top-K queries using a 4-layer iterative self-index is 15.9μs, which is 62.7% lower than the response time without a self-index (42.6μs) and 37.1% lower than that of a single-layer self-index; 3) Storage overhead: The additional storage space occupied by the self-index structure is only 7.8%~9.2% of that of the compressed index, achieving a balance between query performance and storage efficiency.

[0098] The above results verify the effectiveness of the method of the present invention, which can effectively improve the Top-K query performance of massive disk compressed inverted indexes.

[0099] Example 6 This invention presents an iterative self-indexing optimization method for inverted indexes based on the Top-K query model. It aims to solve the challenges of self-index construction caused by variable-length data segments in equal-word compressed inverted indexes, as well as the performance bottlenecks of traditional self-indexes being disconnected from the Top-K query model and exhibiting low random access efficiency. The method first constructs an initial inverted index using open-source tools, converts it to an equal-word compressed format, and reassembles it into a block-interleaved storage structure. Then, based on the Top-K query model, it constructs a multi-layer iterative self-index through recursive segmentation and sampling. Next, it establishes a query performance model including parameters such as disk I / O and decompression speed, quantifies the relationship between query benefits and storage overhead, and solves for the optimal synchronization distance and number of iteration layers. Finally, it achieves fast random access to massive disk-compressed inverted indexes. This invention, through variable-length synchronization distance design, Top-K model parameter optimization, and a multi-layer iterative index structure, significantly improves query efficiency, reduces data reading and decompression overhead, and maintains a Pareto-optimal balance between query performance and storage efficiency under different hardware environments, making it suitable for high-concurrency query scenarios with massive amounts of data.

Claims

1. An inverted index iterative self-index optimization method based on the Top-K query model, characterized in that, The specific steps are as follows: Step 1: By generating a dictionary with compression adaptation attributes, construct an integer sequence type inverted linked list and output the initial inverted index for adapting equal-character compression; Step 2: Convert the initial inverted index built in Step 1 into an equal-word compressed format adapted to the iterative self-index construction; Step 3: Using the inverted index in equal-word compressed format output in Step 2 as input, adapt the compressed data characteristics, storage structure and preliminary segmentation results of Step 2 to establish a Top-K query performance model for the self-index, determine the optimal synchronization distance and iteration layer, and finally form a self-index structure adapted to massive data.

2. The inverted index iterative self-index optimization method based on the Top-K query model as described in claim 1, characterized in that, In step 1, specifically: Step 1.1: Traverse all documents, extract all words contained in each document and summarize them into the same list, perform deduplication on the list to form a dictionary of words without duplicates; Step 1.2: Read the word dictionary list file and the document collection list file with assigned document numbers; Step 1.3: Create an empty word inverted list to store the complete adapted compressed inverted index data. This word inverted list is used to store the entire inverted index. Step 1.4, Outer Loop: Retrieve the current word, initialize a temporary dictionary to store the inverted linked list information of the word, set the 'word' key in the temporary dictionary with the value of the current word, set the 'index' key with the value of an empty list to store the document number containing the word; Step 1.5, Inner Loop: Retrieve the current document information, determine whether the word segmentation list of the current document contains the word of the outer loop, if it does, add the document's index to the 'index' list of the temporary dictionary; if it does not, continue processing the next document. Step 1.6: After completing the inner loop, add the temporary dictionary of the current word to the inverted word list created in Step 1.3; Step 1.7, repeat steps 1.4-1.6 until all entries in the word dictionary have been processed. The final output initial inverted index is the word dictionary and the inverted linked list of words.

3. The inverted index iterative self-index optimization method based on the Top-K query model as described in claim 2, characterized in that, Step 2 specifically involves: Step 2.1: Read the initial inverted index generated in Step 1, extract the inverted linked list corresponding to each term; perform preprocessing on each linked list, verify and ensure that the document number is strictly incremental, convert the document number into a difference sequence, and separate the term frequency sequence, so that each linked list is finally converted into two independent integer sequences: difference and term frequency. Step 2.2: Apply equal-word compression to compress the difference and word frequency integer sequences obtained in Step 2.1 independently. The compression is performed in 32-bit / 64-bit machine words, continuously encoding integers until the available data bits of the current machine word are filled, generating variable-length compressed data segments, and recording the starting address offset of each compressed data segment. Step 2.3 processes two sets of compressed data from the same inverted list: one set is compressed data of document number differences, and the other set is compressed data of the number of times a word appears in the corresponding document, resulting in an inverted index in equal-character compressed format.

4. The inverted index iterative self-index optimization method based on the Top-K query model as described in claim 3, characterized in that, In step 2.3, specifically, the following steps are taken: Two sets of compressed data from the same inverted linked list are processed: one set is compressed data showing the document number differences, and the other set is compressed data showing the frequency of words in their corresponding documents; these two sets of data are then stored alternately together, forming a physical storage block, according to their original order corresponding to each document. Meanwhile, an initial lower limit d for synchronization distance is set, and the entire linked list is logically divided: starting from the beginning of the linked list, the data is decompressed one by one and the number of inverted items is counted. When the cumulative number of inverted items exceeds d for the first time, a mark is made at this position and it is regarded as the logical segment boundary. After all storage and segmentation operations are completed, the final inverted index in equal-word compressed format is obtained.

5. The inverted index iterative self-index optimization method based on the Top-K query model as described in claim 4, characterized in that, Step 3 specifically involves: Step 3.1: Establish a Top-K query performance model with a single-layer self-index to obtain the optimal number of synchronization points; Step 3.2: Based on the Top-K query performance model of a single-layer self-index, add a second-layer self-index structure, derive the two-layer overhead model, and determine the optimal number of synchronization points for the second layer; Step 3.3, Calculation The Top-K query performance model of the layer-iterative self-index structure is used to obtain the optimal synchronization distance and iteration layer number of the self-index structure of each inverted linked list, and finally form a self-index structure adapted to massive data.

6. The inverted index iterative self-index optimization method based on the Top-K query model as described in claim 5, characterized in that, In step 3.1, specifically: Let the length of any inverted list in the inverted index obtained after step 2 be . Suppose a single-level self-index structure contains There are 1 optimal synchronization points; let the synchronization distance be . ; Total query time overhead Includes disk read overhead Data decompression overhead and data search overhead ,Right now The time taken to read each inverted item from disk is , The physical storage block size determined in step 2.3; The read overhead includes the overhead of reading the self-indexed structure generated in step 2 and the physical storage blocks reassembled in step 2, i.e. Let the decompression time for each inverted index be... The decompression overhead includes the overhead of the self-indexed structure generated in decompression step 2 and the overhead of the compressed data segments within the logical segments marked in decompression step 2, i.e. The average lookup cost for a single data segment is the same as that for a single-level self-indexed structure. The search for each synchronization point and the search within the physical storage block reassembled in step 2, i.e. Therefore, the worst-case search performance model is derived as follows: The worst-case search performance model formula is derived by finding the minimum point through differentiation, thus achieving... smallest The value is ; Based on the performance of mainstream servers, the optimal number of synchronization points for a single-level self-indexing structure is inferred. The distribution of synchronization points must be aligned with the boundaries of the compressed data segments generated in step 2. Synchronization points are set according to the data segment length: if the length of the compressed data segment generated in step 2 is less than the lower limit of the synchronization distance... Set the inverted item corresponding to the right boundary of this compressed data segment as the synchronization point; if the length of the compressed data segment generated in step 2 is less than the lower limit of the synchronization distance. This short data segment is then merged sequentially with one or more subsequent compressed data segments until the total number of inverted items in the merged whole exceeds the lower limit of the synchronization distance. Then, set the inverted item corresponding to the right boundary of the merged whole as the synchronization point.

7. The inverted index iterative self-index optimization method based on the Top-K query model as described in claim 6, characterized in that, In step 3.2, specifically: Suppose the second-level self-index structure contains Based on the derivation process in step 3.1, the worst-case query performance model under this self-index structure can be derived as follows: Solving using the above formula makes To obtain the minimum value ( , The combination of these two layers of self-indexing is used to configure the optimal synchronization point for the two layers, and the synchronization distance of the second layer can be calculated.

8. The inverted index iterative self-index optimization method based on the Top-K query model as described in claim 7, characterized in that, In step 3.3, specifically: calculate A Top-K query performance model using a layered iterative self-indexing structure, wherein... To determine the length of the inverted linked list from step 2 and physical storage block size Given a definite theoretical maximum number of layers, let... ;No. The optimal number of synchronization points for a layered self-indexed structure is ; Using the above derivation process, the worst-case query performance model under this self-index structure can be obtained as follows: From the above formula, we can derive that... Minimum of all Values ​​and synchronization distances; by comparing the results under different self-indexing iteration levels. The value determines the optimal number of iteration layers, when When the optimal iteration level of the query performance model is determined, the optimal synchronization distance and iteration level of the self-index structure of each inverted linked list can be obtained, ultimately forming a self-index structure adapted to massive amounts of data.

Images