A data writing method and an asynchronous append file method based on a segmented mmap mechanism

Abstract

The present application relates to the field of data writing, in particular to a data writing method and an asynchronous appending file method based on a segmented mmap mechanism, which comprises the following steps: S1. accepting a parameter CHUNK_SIZE set by a user, and logically equally dividing a file recorded with data by the user according to the parameter; S2. for each chunk of the file, first applying a shared memory equal to the size of CHUNK_SIZE in the shared memory, and attaching the shared memory to the processor space, creating a hole file equal to the size of CHUNK_SIZE, and mapping the file to the memory space applied in step a by calling an mmap function. The present application has the following advantages: compared with mmap of a single file (not scalable), the present application can adapt to an IO stream of a sequential appending type; the mmap is mapped to a shared memory area, so that in the case of process failure, the process can be restarted or a recovery tool is run to check and recover the data not written to the disk; a Manifest metadata file is provided, so that the data to be searched can be quickly indexed to the chunk file.

Description

Technical Field

[0001] This invention relates to the field of data writing, specifically a data writing method and an asynchronous file appending method based on a segmented mmap mechanism. Background Technology

[0002] During the operation of software systems, it is often necessary to record logs or business transaction data. This type of I / O scenario typically involves sequential appending to a file. For example, during the operation of a securities trading system, system logs are output in real time to track the system's operational status. To ensure system recoverability and high reliability, runtime state data and transaction data, such as order transaction data and memory data snapshots, need to be persisted. These operations all involve reading and writing disk files. Existing file writing modes are generally divided into synchronous writing and asynchronous writing. The interface provided by the synchronous writing mode usually requires the caller to wait for the data to be written to the physical storage device before returning. The interface provided by the asynchronous writing mode usually returns immediately after the data to be written is committed, and other processes (threads) can then actually write the data to the physical storage device. The synchronous writing mode returns only after the data is persisted, which maximizes the timeliness and non-volatility of data storage. However, the latency of the synchronous writing interface is strongly correlated with the performance of the physical device, and the write latency of the physical device is usually in the tens of microseconds or even milliseconds.

[0003] Because securities trading systems are extremely latency-sensitive software systems, they typically cannot tolerate near-millisecond latency caused by using synchronous interfaces to record logs and transaction data. Simultaneously, securities trading systems are also extremely sensitive to data consistency, and similarly cannot tolerate the loss of data returned successfully by write operations. Existing technical solutions simply utilize mmap to map files into memory, that is, mapping a file or other object into the process's address space, achieving a one-to-one mapping between the file's disk address and a segment of virtual addresses in the process's virtual address space. Because mmap operations cannot expand file sizes, the existing simple solution of mmapping a single file into memory is unsuitable for sequential appending I / O streams; moreover, existing mmap methods typically map files to ordinary process spaces, making the consistency between memory and files uncontrollable after an abnormal process exit. Summary of the Invention

[0004] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and provide a data writing method and an asynchronous file appending method in a system such as securities trading that is sensitive to low latency and high reliability. This method submits the data to be written to the shared memory buffer of mmap and places the time-consuming data writing task to disk in the kernel background task. This provides the low latency advantage of asynchronous write mode and ensures that even if the process fails, the data returned by the write interface is not lost by rereading the shared memory while the operating system is running normally.

[0005] To achieve the above objectives, a data writing method based on a segmented mmap mechanism is designed. This method includes: S1. Accepting the user-defined parameter CHUNK_SIZE and logically dividing the file containing the user's data into equal parts according to the parameter; S2. For each chunk of the file, first allocating a shared memory space equal to CHUNK_SIZE in shared memory and appending it to the processor space, creating a hollow file equal to CHUNK_SIZE, and mapping this file to the memory space allocated in step a by calling the mmap function; S3. Maintaining three pointers: immutable_chunk: pointing to the previous chunk that was full but not yet persisted; current_chunk: pointing to the currently accepting chunk for writing; next_chunk: the pre-allocated next chunk to accept writing; S4. Maintaining a file named Manifest, which records: The current_chunk corresponds to the file sequence number, and the index information of all chunk files that have been persisted; S5. Run the background disk write thread to write the full chunk data back to the disk, and place the task of pre-allocating new chunks in the background thread to reduce the latency of the data append interface.

[0006] The present invention also has the following preferred technical solutions: 1. The method further includes S6. When current_chunk is full: a. First check if immutable_chunk is empty; if immutable_chunk is not empty, it means that the previous full chunk has not been persisted, and at this time, write a write buffer full error code to suspend or return the write operation; if immutable_chunk is not empty, perform subsequent operations; b. Update the current_chunk sequence number in the Manifest file; c. immutable_chunk = current_chunk, current_chunk = next_chunk; d. Activate the background thread to perform msync of immutable_chunk data and new allocation of next_chunk.

[0007] 2. The index information of all persistent chunk files in step S4: if it is a log file, record the time range of this chunk file; if it is a business transaction record, record the order number range of this chunk file. Since the current_chunk field is an in-place update, the current_chunk field is mmapped into the process space for easy modification, or a separate file CURRENT is created and mmapped into the process space to specifically maintain the current_chunk field.

[0008] An asynchronous append method for the segmented mmap data writing method is also designed. The method is as follows: When opening a logical file, first check if the file's Manifest file exists. If the Manifest file does not exist, it indicates that this is the first time the logical file has been opened, and the corresponding chunk file number should start from the initial value 1. If the Manifest file already exists, it indicates that a logical file has been opened. Regardless of whether the logical file was abnormally closed last time, a data recovery process is performed, and the previously recorded chunk file number n is obtained. New chunk files are created starting from the current file number and mapped to memory space. When current_chunk is full, check if the immutable_chunk pointer is null: If immutable_chunk is pointing to a chunk address space, it means that the previously full chunk has not yet been persisted to the physical storage device. In this case, append operations should be suspended or a write buffer full error should be returned. If immutable_chunk is NULL, it means that the previously full chunk has been written to disk. In this case, immutable_chunk is set to point to current_chunk. Set current_chunk to next_chunk and activate a background thread to perform the following: a. Create a new empty file of CHUNK_SIZE, mmap it to the shared space, attach it to the process address, and let next_chunk point to it; b. Perform msync on immutable_chunk, write the data back to the device, and release the shared memory space; c. Update the index information of immutable_chunk to the Manifest file; d. set immutable_chunk = NULL.

[0009] Preferably, the data recovery process is as follows: the recovery process restores the data remaining in the shared memory area after the process exits due to a failure to a file. Since the remaining data is in the shared memory area, the recovery process is executed during the process restart or runs independently as a separate recovery tool.

[0010] Preferably, the data recovery process is as follows: a. First, obtain the current_chunk number I at the time of the last exit through the Manifest file; b. chunk{i+1} is a pre-allocated chunk, so if it is stored in the chunk{i+1} file, it is directly deleted; if there is shared memory mapped to the chunk{i+1} file, it is directly released; c. chunk{i-1} is the largest chunk that is full. If the shared memory corresponding to chunk{i-1} has been released, it means that the immutable_chunk has completed msync before the process failed or exited; if the memory of chunk{i-1} does not exist, it means that the immutable_chunk has not completed msync before the process exited. It is necessary to compare the data in the shared memory with the data in the chunk{i-1} file, and write back the data that has not been written to disk, or write the data in the shared memory to a new file to replace the original chunk{i-1} file; d. The chunk{i} file is the current_chunk at the time of the process exit, and the recovery method is the same as chunk{i-1}.

[0011] Compared with the prior art, the advantages of this invention are: 1. Compared to mmap (non-expandable) for a single file, this invention can adapt to sequential append-type IO streams; 2. Mapping mmap to a shared memory area allows for the verification and recovery of data not written to disk in the event of a process failure by restarting the process or running recovery tools; 3. Provides a Manifest metadata file, which can quickly index which Chunk file the data to be searched is located in. Attached Figure Description

[0012] Figure 1 This is a structural diagram of the segmented mmap of the present invention; Figure 2 This is an additional flowchart of the present invention document; Figure 3 This is a flowchart of the recovery process of the present invention. Detailed Implementation

[0013] The invention will be further described below with reference to the accompanying drawings. The structure and principle of the invention are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0014] like Figure 1 As shown, the data writing methods of the segmented mmap mechanism include: 1. The user-input parameter CHUNK_SIZE will logically divide a file containing appended user records into equal parts, for example, CHUNK_SIZE = 1GB; 2. For each chunk of the file, it will: a. First, allocate a shared memory space of size CHUNK_SIZE in the shared memory and attach it to the process space; b. Create a hole file of size CHUNK_SIZE; c. Map the file to the memory space allocated in step a using mmap; 3. Maintain 3 pointers, a.immutable_chunk: Points to the previous chunk that is full but not yet persisted; b.current_chunk: Points to the chunk that is currently being written to; c.next_chunk: The pre-allocated next chunk to accept writes; 4. Maintain the Manifest file to record: a. The file number corresponding to the current_chunk; b. Index information for each persisted chunk file; (e.g., if it's a log file, the time range of this chunk file can be recorded; if it records business transaction logs, the order number range of this chunk file can be recorded). Since the current_chunk field is an in-place update, it can be mmapped into the process space for easy modification, or a separate file, CURRENT, can be created and mmapped into the process space to specifically maintain the current_chunk field. If the latter is used, the Manifest file only needs to maintain the content in b.

[0015] 5. Background disk write thread: Time-consuming operations such as writing data from a full chunk back to disk and pre-allocating new chunks are performed in the background thread to minimize the latency of calling the append data interface. 6. When current_chunk is full, a. First, check if immutable_chunk is empty; if immutable_chunk is not empty, it means that the previous full chunk has not been persisted. At this time, the write operation will be suspended or a write buffer full error code will be returned; if immutable_chunk is not empty, subsequent operations can be performed. b. Update the current_chunk sequence number in the Manifest file; c.immutable_chunk = current_chunk; current_chunk = next_chunk; d. Activate background threads to perform msync on immutable_chunk data and new allocation of next_chunk, etc.

[0016] like Figure 2 As shown, the flowchart for opening a logic file and appending data is as follows: When opening a logical file, first check if the file's Manifest file exists. If the Manifest file does not exist, it means that this is the first time the logical file has been opened, and the corresponding chunk file number should start from the initial value 1. If the Manifest file already exists, it means that a logical file has been opened. Regardless of whether the logical file was closed abnormally last time, a data recovery process can be performed to obtain the chunk file number n recorded last time. Starting from the current file number, new chunk files are created and mapped to memory space. When `current_chunk` is full, check if the `immutable_chunk` pointer is null: If `immutable_chunk` is pointing to a chunk address space, it means the previously full chunk has not yet been persisted to the physical storage device. In this case, append operations should be suspended or a write buffer full error should be returned. If `immutable_chunk` is NULL, it means the previously full chunk has been written to disk (guaranteed by a background thread). In this case, `immutable_chunk` is set to point to `current_chunk`, `current_chunk` is set to `next_chunk`, and a background thread is activated to proceed. a. Create a new hole file of size CHUNK_SIZE, mmap it to the shared space and attach it to the process address, pointed to by next_chunk; b. Perform msync on the immutable_chunk, write the data back to the device, and release the shared memory space; c. Update the index information of immutable_chunk to the Manifest file; d.immutable_chunk = NULL; like Figure 3 As shown, the recovery process is as follows: The recovery process can restore data remaining in the shared memory area after a process crashes and exits to a file.

[0017] Because the legacy data is in the shared memory area, the recovery process can be executed during process restart or run independently as a separate recovery tool.

[0018] First, obtain the current_chunk sequence number i from the last exit using the Manifest file. 1. chunk{i+1} is a pre-allocated chunk, so if it is stored in the chunk{i+1} file, it can be deleted directly; if there is shared memory mapped to the chunk{i+1} file, it can be released directly. 2. chunk{i-1} is the largest chunk that is already full. If the shared memory corresponding to chunk{i-1} has been released, it means that immutable_chunk has completed msync before the process crashed or exited. If the memory of chunk{i-1} does not exist, it means that immutable_chunk has not completed msync before the process exited. It is necessary to compare the data in the shared memory with the data in the chunk{i-1} file and write back the data that has not been written to disk (or write the data in the shared memory to a new file to replace the original chunk{i-1} file). 3. The chunk{i} file is the current_chunk when the process exits, and it is restored in the same way as chunk{i-1}.

[0019] The above description is merely a specific embodiment of the invention, but the scope of protection of the invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the technical scope disclosed in the invention, based on the technical solutions and novel concepts of the invention, should be covered within the scope of protection of the invention.

Claims

1. A data writing method based on a segmented mmap mechanism, characterized in that, The method includes: S1. Accept the user-defined parameter CHUNK_SIZE and perform logical equal-size division on the file containing the user's recorded data according to the parameter; S2. For each chunk of the file, first allocate a shared memory space equal to CHUNK_SIZE in the shared memory and attach it to the processor space, create an empty file equal to CHUNK_SIZE, and map this file to the memory space allocated in step a by calling the mmap function; S3. Maintain three pointers. immutable_chunk: Points to the previous chunk that was full but not yet persisted; current_chunk: Points to the chunk that is currently being written to; next_chunk: The pre-allocated next chunk to accept writes; S4. Maintain a file named Manifest, which records: the file sequence number corresponding to the current_chunk, and the index information of all persistent chunk files; wherein, the index information is: if the file is a log file, the time range of the corresponding chunk file is recorded; if the file is a business transaction file, the order number range of the corresponding chunk file is recorded; the current_chunk field is maintained in any of the following ways: (a) mmap the current_chunk field to the process space for easy modification; (b) create a separate file CURRENT and mmap it to the process space to specifically maintain the current_chunk field; S5. Run a background disk write thread to write the full chunk data back to the disk, and place the task of writing the new pre-allocated chunk in the background thread to reduce the latency of the data append interface. S6. When current_chunk is full: a. First, check if immutable_chunk is empty; if immutable_chunk is not empty, it means that the previous full chunk has not been persisted. At this time, the write operation will be suspended or a write buffer full error code will be returned; if immutable_chunk is not empty, proceed with the subsequent operations. b. Update the current_chunk sequence number in the Manifest file; c.immutable_chunk = current_chunk, current_chunk = next_chunk; d. Activate a background thread to perform msync on the immutable_chunk data and allocate the new next_chunk.

2. The data writing method based on segmented mmap mechanism as described in claim 1, characterized in that... In step S4, the index information of all persistent chunk files is recorded. If it is a log file, the time range of this chunk file is recorded; if it is a business transaction record, the order number range of this chunk file is recorded. Since the current_chunk field is an in-place update, the current_chunk field is mmap'd into the process space for easy modification, or a separate file CURRENT is created and mmap'd into the process space to specifically maintain the current_chunk field.

3. An asynchronous file appending method using the segmented mmap data writing method as described in claim 1 or 2, characterized in that... The method is as follows: When opening a logical file, first check if the file's Manifest file exists. If the Manifest file does not exist, it means that this is the first time the logical file has been opened, and the corresponding chunk file number should start from the initial value 1. If the Manifest file already exists, it means that a logical file has been opened. Regardless of whether the logical file was closed abnormally last time, a data recovery process is performed, and the previously recorded chunk file number n is obtained. Starting from the current file number, new chunk files are created and mapped to memory space. When current_chunk is full, check if the immutable_chunk pointer is null: if immutable_chunk is pointing to a chunk address space, it means that the previously full chunk has not yet been persisted to the physical storage device. In this case, append operations should be suspended or a write buffer full error should be returned; if immutable_chunk is NULL, it means that the previously full chunk has been written to disk. In this case, immutable_chunk is set to current_chunk, current_chunk is set to next_chunk, and a background thread is activated to perform the operation. a. Create a new hole file of size CHUNK_SIZE, mmap it to the shared space and attach it to the process address, pointed to by next_chunk; b. Perform msync on the immutable_chunk, write the data back to the device, and release the shared memory space; c. Update the index information of immutable_chunk to the Manifest file; d.immutable_chunk = NULL.

4. The asynchronous append file method as described in claim 3, characterized in that... The data recovery process is as follows: The recovery process restores the data remaining in the shared memory area after the process exits due to a failure to a file. Since the remaining data is in the shared memory area, the recovery process is executed during the process restart or runs separately as an independent recovery tool.

5. The asynchronous append file method as described in claim 4, characterized in that... The data recovery process is as follows: a. First, obtain the current_chunk sequence number I from the last exit using the Manifest file; b.chunk{i+1} is a pre-allocated chunk, so if it is stored in the chunk{i+1} file, it will be deleted directly; if there is shared memory mapped to the chunk{i+1} file, it will be released directly. c.chunk{i-1} is the largest chunk that is already full. If the shared memory corresponding to chunk{i-1} has been released, it means that immutable_chunk has completed msync before the process crashed or exited. If the memory of chunk{i-1} does not exist, it means that immutable_chunk has not completed msync before the process exited. It is necessary to compare the data in the shared memory with the data in the chunk{i-1} file and write back the data that has not been written to disk, or write the data in the shared memory to a new file to replace the original chunk{i-1} file. The d.chunk{i} file is the current_chunk when the process exits, and it is restored in the same way as chunk{i-1}.

Images