IO merging mechanism based on block index

Abstract

The invention discloses an IO merging mechanism based on a block index, relates to the technical field of big data and information retrieval, and optimizes an original multi-file stream output mechanism of open source Lucene into single-file stream output by expanding a Directory interface of the Lucene, so that IO operation merging is realized in an index writing process, and the operation efficiency is improved. The method specifically comprises three parts of a file storage structure, a writing process and a reading process. The file storage structure comprises a single file structure of a file header, a data area and a file tail; during writing, data is cached in the memory block firstly, and after the memory block is full, the data is added to a file by taking a data block as a unit; during reading, quick positioning is realized by analyzing the block index at the end of the file; according to the method, the problems of performance bottleneck, small file flooding, difficulty in atomicity guarantee and the like caused by multi-file IO in the prior art are solved, and the index construction efficiency and the system stability are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of big data and information retrieval technology, and in particular to an IO merging mechanism based on block indexes. Background Technology

[0002] In modern search engines and big data systems, Apache Lucene is the most widely used open-source full-text search engine. Lucene divides index data into multiple independent physical files, each corresponding to a different index structure, and performs unified file I / O operations through the Directory abstract interface. The open-source Lucene storage structure is as follows: Figure 1 As shown.

[0003] What are the shortcomings or disadvantages of existing technologies?

[0004] Frequent and scattered I / O operations: Each write or refresh operation requires the creation of multiple independent files, resulting in a large amount of random I / O. Especially in high-concurrency write scenarios, the I / O bottleneck is severe, significantly reducing index building efficiency.

[0005] The number of files explodes: Frequent index refresh and merge operations will generate a massive number of small files on the disk, which not only increases the file system management overhead, but also causes the "small file problem", affecting the performance and stability of the file system.

[0006] High resource overhead: Persistent data requires each file to be opened, closed, and its file handle to be maintained individually. The large number of file handles leads to system resource shortages, especially in containerized or microservice architectures.

[0007] The fault tolerance and recovery mechanisms are complex: When write operations are interrupted, especially in HDFS distributed storage environments, ensuring the atomicity of multiple independent files is difficult, easily leading to inconsistent states where some files are written successfully while others fail, increasing the difficulty of data recovery. To alleviate these problems, existing technologies attempt to package multiple independent files into a single composite file by merging index segments (e.g., performing segment merging during IndexWriter.commit()) or using the Compound File System (CFS) mechanism. However, CFS essentially still relies on the encapsulation of the original multi-file structure and requires rereading and merging the contents of each file after the write operation is complete, belonging to the "write-after-merge" mode. This approach cannot achieve aggregation and optimization of I / O operations during the index write phase, making it difficult to fundamentally reduce frequent random I / O during the write process. Especially in near real-time retrieval scenarios with high concurrency and high-frequency refresh, the I / O performance bottleneck remains significant.

[0008] Therefore, there is an urgent need in this field for a technical solution that can achieve IO merging during the index writing stage, thereby effectively solving the aforementioned performance and resource bottlenecks. Summary of the Invention

[0009] The technical problem to be solved by this invention is to provide an I / O merging mechanism based on block index to address the shortcomings of the prior art. By extending the Directory interface of Lucene, a new single-file streaming storage structure is designed to merge the originally scattered writing of multiple files into sequential append writing of a single physical file.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0011] An I / O merging mechanism based on block index is proposed. By extending Lucene's Directory interface, the original multi-file stream output mechanism of open-source Lucene is optimized into single-file stream output, thereby realizing the merging of I / O operations during the index writing process. Specifically, it includes three parts: file storage structure, writing process and reading process.

[0012] Directory is an interface in Lucene used to abstract file system operations, defining basic operations such as file creation, reading, and deletion.

[0013] As a further preferred embodiment of the block index-based IO merging mechanism of the present invention, the file storage structure includes a file header, a data area, and a file footer.

[0014] The file header (Header) is located at the beginning of the file and is used to store format identifiers, version information, and metadata.

[0015] Data section: The main body of the file, used to store all the data from the original Lucene index files;

[0016] Footer: Located at the end of the file, it stores metadata and checksum data required for parsing the file.

[0017] As a further preferred embodiment of the block index-based IO merging mechanism of the present invention, the write process

[0018] Specifically, it includes the following three steps:

[0019] Step 1, Initialization: Create the target file, open the file output stream, and generate and write the file header information;

[0020] Step 2, Data Writing: During the index writing process, when the flush condition is met, DirectoryObjectWriter will be called to perform data persistence operations;

[0021] Step 3, Write complete: Generate and write file end information, close file output stream; in case of exception, ensure cleanup logic is executed to release all occupied resources and ensure system stability and data integrity.

[0022] As a further preferred embodiment of the block index-based IO merging mechanism of the present invention, the data writing in step 2 is specifically implemented as follows:

[0023] Step 2.1: Create a wrapped IndexOutput object. This object does not directly create physical files, but serves as a logical interface for data writing. Multiple IndexOutput instances are usually created in parallel to support multi-threaded or parallel writing and improve writing efficiency.

[0024] Among them, IndexOutput is an abstract class in Lucene used for writing index data. It implements the core logic of writing a single file stream by encapsulating ObjectChunkIndexOutput. As a logical writing unit, it temporarily stores the original index data in a buffer and appends it to a unified physical file in blocks, thereby achieving efficient conversion from multi-file writing to single-file stream output.

[0025] Step 2.2: Each IndexOutput holds a ChunkBuffer to temporarily buffer the data to be written; all write operations are first written to the buffer to avoid frequent disk I / O.

[0026] Step 2.3: When the buffer reaches the preset capacity, the entire buffer is appended to the index file as a complete DATA_CHUNK unit, and the buffer is cleared to continue subsequent writing; DATA_CHUNK is a data block in the data area divided into fixed sizes, which serves as the basic unit for writing;

[0027] In step 2.4, each IndexOutput records the offset of each CHUNK append operation. After writing, this offset information will be used to construct CHUNK_INDEX, enabling efficient location and fast access to the index data.

[0028] As a further preferred embodiment of the block index-based IO merging mechanism of the present invention, the reading process specifically includes the following two steps:

[0029] Step S1, Initialization: Before executing the index query, first open the IndexReader and complete the initialization of the DirectoryObjectWriter;

[0030] Step S2, Index Query: When performing an index query, data is read using DirectoryObjectReader.

[0031] As a further preferred embodiment of the block index-based IO merging mechanism of the present invention, in step S1, the following operations are initialized:

[0032] (1) Open and maintain the input stream of the index file;

[0033] (2) Read the footer information at the end of the file;

[0034] (3) Parse CHUNK_INDEX.

[0035] As a further preferred embodiment of the IO merging mechanism based on block index of the present invention, step S2, the specific implementation steps of index query are as follows:

[0036] (1) Using the file input stream that has been opened by DirectoryObjectWriter, and the offset of the corresponding index file in CHUNK_INDEX, create a new ObjectChunkIndexInput object;

[0037] (2) Data is read by relative position through ObjectChunkIndexInput. The actual data position is calculated based on the offset information inside the input, thus completing the data reading.

[0038] (3) After the data reading is complete, only the ObjectChunkIndexInput object itself is released, without closing the underlying index file input stream.

[0039] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0040] 1. This invention significantly reduces I / O operations and improves index write performance: Open-source Lucene uses a multi-file write mechanism, which requires the creation of multiple independent files for each index refresh, resulting in a large number of random I / O operations; This invention merges the writing of multiple index files into a sequential stream write to a single physical file, and reduces the number of I / O operations by more than 90% through ChunkBuffer buffering and DATA_CHUNK batch append mechanism. Especially in high-concurrency write scenarios, the throughput of index building is significantly improved, effectively alleviating the I / O bottleneck;

[0041] 2. This invention completely solves the "small file problem" and reduces file system overhead: Traditional write mode easily generates hundreds or thousands of small index files, causing problems such as high file system metadata pressure, large file handle occupation, and frequent GC; This invention writes all index data into a single physical file, fundamentally eliminating the "small file problem", significantly reducing the file system management burden and memory resource consumption, and is particularly suitable for distributed file systems;

[0042] 3. This invention achieves I / O merging during the write phase: Existing technologies employ a "write-after-merge" strategy, which involves writing multiple files first and then merging them into a composite file, failing to optimize I / O during the write process. This invention pioneers a merging mechanism during the write process, aggregating multiple file writes into a single file stream output from the source, achieving true I / O merging, and representing a fundamental breakthrough in index write performance optimization.

[0043] 4. This invention has good scalability and compatibility, and supports multiple version evolution: This invention introduces the CODE_VERSION field in the file header, which supports multiple version format compatibility and can achieve backward compatibility and smooth upgrade; the structural design reserves extension fields to facilitate future support for advanced features and has good technical extensibility. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the open-source Lucene storage structure;

[0045] Figure 2 This is a schematic diagram of the implementation logic before optimization;

[0046] Figure 3 This is a schematic diagram of the optimized implementation logic;

[0047] Figure 4 This is a schematic diagram of the index file storage structure;

[0048] Figure 5 This is a schematic diagram of the writing process of this invention;

[0049] Figure 6 This is a schematic diagram of the query process of this invention;

[0050] Figure 7 These are images showing the storage effects before and after optimization according to the present invention;

[0051] Figure 8 This is a diagram showing the optimized storage effect of the present invention. Detailed Implementation

[0052] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0054] This invention aims to solve the following key problems existing in the current Lucene index writing mechanism:

[0055] Traditional multi-file writing leads to frequent I / O and performance bottlenecks: The existing multi-file stream output mechanism of Lucene based on the Directory interface causes each index write to trigger multiple independent file creation and write operations, resulting in a large amount of random I / O, which makes it difficult to meet the requirements of high throughput and low latency index writing.

[0056] The surge in the number of files leads to increased system resource consumption and management complexity: the presence of a large number of small files exacerbates file system metadata pressure, handle occupation, and GC burden, especially in large-scale index building scenarios, which reduces system stability;

[0057] The write process lacks atomicity and fault tolerance guarantees: If an abnormal interruption occurs during the writing of multiple files, some files may be written successfully while others fail, resulting in index inconsistency and making it difficult to achieve "all or nothing" atomic writing.

[0058] This invention extends Lucene's Directory interface by introducing DirectoryObjectWriter, optimizing the original multi-file stream output mechanism of open-source Lucene into a single-file stream output, thereby merging I / O operations during index writing. Simultaneously, it introduces DirectoryObjectReader to adapt to single-file reading logic. For a detailed comparison of the I / O implementation logic before and after optimization, please refer to [link to relevant documentation]. Figure 2 , Figure 3 Directory is the interface in Lucene used to abstract file system operations, defining basic operations such as file creation, reading, and deletion.

[0059] IndexOutput is an abstract class in Lucene used for writing index data. In this invention, the core logic for writing a single file stream is implemented by encapsulating ObjectChunkIndexOutput. As a logical writing unit, the original index data is temporarily stored in a buffer and appended to a unified physical file in blocks, thereby achieving efficient conversion from multi-file writing to single-file stream output.

[0060] IndexInput is an abstract class in Lucene used for reading index data. In this invention, ObjectChunkIndexInput is used to achieve efficient positioning and reading of a single file stream.

[0061] By extending Lucene's Directory interface, a new single-file streaming storage structure is designed to merge the originally scattered multi-file writing into sequential append writing to a single physical file.

[0062] The technical solution of the present invention specifically includes the following parts:

[0063] 1. File storage structure:

[0064] In scenarios using single-file stream output, a new index file storage structure needs to be designed. The overall structure consists of three parts: Header, Data, and Footer, as follows: Figure 4 As shown.

[0065] (1) Header: The file header is used to record format identifiers and related metadata, and includes the following fields:

[0066] MAGIC_CODE_HEAD: A fixed identifier (i.e., magic number) in the file header, used to identify the file type and for validity verification;

[0067] CODE_FORMAT and CODE_VERSION: These represent the file's encoding format and version information, respectively. They support a multi-version compatibility mechanism to ensure readability and backward compatibility between different versions.

[0068] (2) Data: The data area adopts a block storage method, which divides the original Lucene file into multiple DATA_CHUNKs of fixed size and writes them to the file in the order of disk writing.

[0069] (3) Footer: The footer portion of the file is organized in the following order:

[0070] CHUNK_SIZE: Records the size of each data block (DATA_CHUNK), used to parse the data area structure; where DATA_CHUNK is a data block divided into fixed sizes in the data area, serving as the basic unit for writing;

[0071] CHUNK_INDEX: Maintains the mapping relationship between the original Lucene file and each DATA_CHUNK. See details for the specific structure. Figure 4 ;

[0072] FOOTER_CHECKSUM: CRC checksum at the end of the file, used to verify file integrity;

[0073] FOOTER_LENGTH: The effective length of the file's end, used to quickly locate the start position of the file's end;

[0074] FOOTER_MAGIC_CODE: A fixed identifier (magic number) for the end of a file, used to verify the validity of the end of the file.

[0075] 2. Writing process

[0076] The writing process is as follows Figure 5 As shown, it mainly includes the following three steps:

[0077] (1) Initialization:

[0078] Before writing to the index, the DirectoryObjectWriter must first be initialized. This stage mainly involves the following operations: creating the target file, opening the file output stream, and generating and writing the file header information to prepare for subsequent data writing.

[0079] (2) Data writing:

[0080] During index writing, when the flush condition is met, the system will call DirectoryObjectWriter to perform data persistence operations. The specific implementation is as follows:

[0081] Create a wrapped IndexOutput object. This object does not directly create physical files, but serves as a logical interface for data writing. Multiple IndexOutput instances are usually created in parallel to support multi-threaded or parallel writing and improve writing efficiency.

[0082] Each IndexOutput holds a ChunkBuffer to temporarily buffer data to be written; all write operations are first written to the buffer to avoid frequent disk I / O.

[0083] When the buffer reaches the preset capacity, the entire buffer is appended to the index file as a complete CHUNK unit, and the buffer is cleared to continue subsequent writing;

[0084] Each IndexOutput records the offset of each CHUNK append operation. After the write is complete, this offset information will be used to construct CHUNK_INDEX, enabling efficient location and fast access to the index data.

[0085] (3) Writing complete:

[0086] After the index is written, the close method of DirectoryObjectWriter is called to perform the closing operation. This stage mainly completes the following operations: generating and writing the end-of-file information, closing the file output stream; and ensuring that cleanup logic is executed in case of exceptions, releasing all occupied resources, and ensuring system stability and data integrity.

[0087] 3. Reading process

[0088] The reading process is as follows Figure 6 As shown, it mainly consists of two steps:

[0089] (1) Initialization: Before executing an index query, the system first opens the IndexReader and completes the initialization of the DirectoryObjectWriter. This stage mainly involves the following operations:

[0090] Open and maintain the input stream for the index file;

[0091] Read the footer information at the end of the file;

[0092] Parse CHUNK_INDEX.

[0093] (2) Index lookup

[0094] When performing an index query, data is read using DirectoryObjectReader, as implemented below:

[0095] Using the file input stream already opened by DirectoryObjectWriter, and combining it with the offset of the corresponding index file in CHUNK_INDEX, create a new ObjectChunkIndexInput object;

[0096] Data is read by relative position through ObjectChunkIndexInput, which internally calculates the actual data position based on the offset information to complete the data reading.

[0097] After data reading is complete, only the ObjectChunkIndexInput object itself is released; the underlying index file input stream is not closed. The query process is as follows: Figure 6 As shown

[0098] Example:

[0099] Detailed explanation of the key processes of the IO merging mechanism:

[0100] 1. Write implementation

[0101] (1) Write initialization

[0102]

[0103] (2) Data writing

[0104]

[0105] 2. Reading Implementation (1) Initialization

[0106]

[0107] (3) Data reading

[0108]

[0109] 3. Comparison of storage performance before and after optimization: Before optimization, as shown below: Figure 7 As shown in the figure, after optimization... Figure 8 As shown.

[0110] This invention is based on a single-file stream index writing architecture: an "IO merging" mechanism is introduced in the Lucene index writing stage to merge the originally scattered multi-file writing into a unified sequential stream writing, fundamentally reducing the I / O frequency and the number of files.

[0111] It adopts a three-segment storage structure of "Header + Data + Footer";

[0112] Chunk buffering and append-only mechanism: Each IndexOutput holds an independent ChunkBuffer. After buffering and writing, the data is appended to the end of the file in units of DATA_CHUNK. Parallel writing and dynamic merging are supported.

[0113] The DirectoryObjectReader parses the CHUNK_INDEX at the end of the file, creates an ObjectChunkIndexInput object, and achieves efficient positioning and reading of the original index data based on the offset.

[0114] Atomicity and fault tolerance guarantee mechanism: The write strategy of "write data first, then write footer, and commit as a whole" is adopted, combined with magic number and check code to ensure the atomicity of the write process and the integrity of the file.

[0115] Compared to existing technologies, this invention significantly reduces the number of I / O operations and substantially improves index write performance. Open-source Lucene employs a multi-file write mechanism, requiring the creation of multiple independent files for each index refresh, resulting in numerous random I / O operations. This invention merges the writing of multiple index files into a sequential stream write to a single physical file. Through ChunkBuffer buffering and DATA_CHUNK batch appending mechanism, the number of I / O operations is reduced by more than 90%. Especially in high-concurrency write scenarios, the index building throughput is significantly improved, effectively alleviating the I / O bottleneck.

[0116] Completely solves the "small file problem" and reduces file system overhead: Traditional write modes easily generate hundreds or thousands of small index files, leading to problems such as high file system metadata pressure, excessive file handle usage, and frequent garbage collection. This invention writes all index data into a single physical file, fundamentally eliminating the "small file problem," significantly reducing the file system management burden and memory resource consumption, and is particularly suitable for distributed file systems.

[0117] Achieving I / O merging during the write phase: Existing technologies employ a "write-after-merge" strategy, which involves writing multiple files first and then merging them into a composite file, failing to optimize I / O during the write process. This invention pioneers a merging mechanism during the write process, aggregating multiple file writes into a single file stream output from the source, achieving true I / O merging, and representing a fundamental breakthrough in index write performance optimization.

[0118] It possesses excellent scalability and compatibility, supporting multiple version evolutions: This invention introduces CODE_VERSION in the file header.

[0119] The fields support multiple version format compatibility, enabling backward compatibility and smooth upgrades; the structural design reserves extended fields to facilitate future support for advanced features, possessing good technical extensibility.

[0120] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0121] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A block index based IO merging mechanism, characterized in that: The original multi-file stream output mechanism of the open source Lucene is optimized to single file stream output by extending the Directory interface of Lucene, so that the merging of IO operations is realized in the index writing process, and the file storage structure, the writing process and the reading process are included. The Directory is an interface for abstracting file system operations in Lucene, and is used to define the basic operations of file creation, reading and deletion.

2. The block index based IO consolidation mechanism of claim 1, wherein: The file storage structure includes a file header Header, a data area Data and a file tail Footer. The file header Header is located at the beginning of the file, and is used to store format identification and version information metadata. The data area Data is the main part of the file, and is used to store the data of all original Lucene index files. The file tail Footer is located at the end of the file, and is used to store the meta information and check data required for parsing the file.

3. The block index based IO consolidation mechanism of claim 1, wherein: The writing process specifically includes the following three steps: Step 1, initialization: creating a target file, opening a file output stream, and generating and writing file header information; Step 2, data writing: in the index writing process, when the flush condition is met, the DirectoryObjectWriter is called to perform data persistence operation; Step 3, writing completion: generating and writing file tail information, closing the file output stream; in the event of an exception, ensure that the cleaning logic is executed, and all occupied resources are released, so as to guarantee the system stability and data integrity.

4. The block index based IO consolidation mechanism of claim 3, wherein: In step 2, the data writing is specifically implemented as follows: Step 2.1, an encapsulated index output object IndexOutput is created, which is not directly created as a physical file, but as a logical interface for data writing; usually, multiple IndexOutput instances are created in parallel to support multi-threaded or parallel writing and improve writing efficiency; IndexOutput is an abstract class for writing index data in Lucene, which realizes the core logic of single file stream writing through encapsulating ObjectChunkIndexOutput, and serves as a logical writing unit to temporarily store the original index data in the buffer and append it to the unified physical file in blocks, so as to realize the efficient conversion of multi-file writing to single file stream output; Step 2.2, each IndexOutput holds a block buffer ChunkBuffer for temporarily buffering the data to be written; all writing operations are first written to the buffer to avoid frequent disk IO; Step 2.3, when the buffer reaches the preset capacity, the entire buffer is appended to the index file as a complete DATA_CHUNK unit, and the buffer is emptied to continue subsequent writing; DATA_CHUNK is a data block divided in the data area according to a fixed size, and is used as a basic unit of writing; Step 2.

4. Each IndexOutput records the offset of each chunk appending operation. After the writing is completed, the offsets are used to build the CHUNK_INDEX, which enables efficient positioning and fast access to the index data.

5. The block index based IO consolidation mechanism of claim 1, wherein: The reading process specifically includes the following two steps: Step S1, initialization: Before performing the index query, first open the IndexReader, and the initialization of DirectoryObjectWriter needs to be completed first; Step S2, index query: When performing the index query, data reading is completed through DirectoryObjectReader.

6. The block index based IO consolidation mechanism of claim 5, wherein: In step S1, the initialization performs the following operations: (1) Open and maintain the input stream of the index file; (2) Read the Footer information at the end of the file; (3) Parse the CHUNK_INDEX.

7. The block index based IO consolidation mechanism of claim 5, wherein: Step S2, the index query specifically implements the following steps: (1) Use the file input stream opened by DirectoryObjectWriter, combined with the offset of the corresponding index file in CHUNK_INDEX, to create a new ObjectChunkIndexInput object; (2) Read the data by ObjectChunkIndexInput according to the relative position, which internally calculates the actual data position according to the offset information, thereby completing the data reading; (3) After the data reading is completed, only the ObjectChunkIndexInput object itself is released, and the underlying index file input stream is not closed.

Images