Design and implementation of efficient log storage

Abstract

Embodiments of the present disclosure relate to the design and implementation of efficient log storage. Methods and systems are disclosed for a log engine that records data operations in a file system. The log engine includes a user interface module configured to receive a plurality of data operation logs recording data operations and sequence IDs corresponding to an order of the data operations being performed, respectively; and a shared log manager configured to provide an in-queue configured to integrate a plurality of data operation logs ordered based on a sequence ID and saved in a shared log, and to exchange the plurality of data operation logs to an out-queue.

Description

Technical Field

[0001] The present disclosure relates to key-value storage and file systems. In particular, the present disclosure relates to a log engine for managing log data for key-value storage and file systems. Background Art

[0002] Log data is used in key-value stores, such as existing database systems that use log-structured merge trees. These provide fast data ingestion, but may come with tradeoffs in write amplification, resource contention, relatively poor read performance, and computational overhead. Summary of the invention

[0003] The present disclosure relates to key-value storage and file systems. In particular, the present disclosure relates to a log engine for managing log data for key-value storage and file systems.

[0004] Log data can be included in a data center log file to record data operations. Log files can include logs of data operations, logs of metadata, etc. In some embodiments, log data can be any time series data, including scientific or experimental data over time, financial or economic data over time, and any other data being received and recorded and its sequence information. Log data can be implemented as an append-only file, which adds the additional log data only to the end of the data.

[0005] To achieve data consistency and integrity, each data operation in the log engine can be assigned a unique sequence ID, allowing data to be arranged on the disk in order according to the sequence ID. This sorting mechanism can support error recovery and maintain the reliability of log files used in upper-level software applications. In addition, the sorting mechanism and / or stores log data in a predefined file size in a continuous memory allocation (for example, by requesting to expand the memory allocation), eliminating or reducing random writes on the SSD. The reduction in random writes to the SSD reduces tail latency and improves the efficiency of managing file metadata updates in memory. Operations can be sorted using and according to a unique sequence ID to ensure order on the disk (such as an SSD). High reliability is achieved through cyclic redundancy check (CRC) protection of the log file. The redundant check protection provided by the log engine can achieve error recovery in the upper-level software. It should be understood that the log engine may include CRC protection for error detection in the log file, which supports error recovery and improves reliability. In one embodiment, a shared log manager (for example, Figure 4 420) can be configured to perform CRC on the log file.

[0006] By aggregating operations from multiple users into a shared log to be processed in one I / O operation, the log engine can handle the high throughput requirements of data operations while reducing write amplification and improving storage efficiency. At the same time, by adapting the shared log, the log engine can use lock-free queues to minimize thread blocking, thereby achieving high concurrency and better resource utilization.

[0007] The log engine can reduce memory overhead by including a background thread to periodically save in-memory data (e.g., log data and / or metadata, etc.) to the underlying file system when the in-memory data reaches a certain size. This approach allows the data memory usage ratio (e.g., the percentage of memory usage for log files to the total memory size) to be as low as 0.1%, allowing efficient memory management and freeing up resources for other tasks.

[0008] The log file size (e.g., metadata log file, operation log file, etc.) is implicitly fixed by predefining the file size. For example, when a new storage extension is allocated to the data to accommodate a larger file size, metadata related to the file size is synchronized. The frequency of requesting and allocating a new extension or the next extension is relatively low compared to the total number of synchronizations of append-only files (e.g., log files). Typically, the synchronization function triggers two separate input / output (I / O) operations: one for updating the data and the other for updating the metadata of a particular file, resulting in at least two I / O operations for each synchronization operation. By reducing or eliminating the need to update the file size, in some embodiments, the write amplification factor (WAF) of the logging engine can be reduced by 1, substantially improving the efficiency of the system.

[0009] It should be understood that the present disclosure describes a log engine for log data about a log engine for a file system. However, the embodiments of the log engine disclosed in this application are considered to be illustrative in all respects and are not limited to file system applications. For example, a log engine for recording time series data can be a log engine for recording scientific or experimental data over time (e.g., temperature, pressure, concentration, etc.), financial or economic data over time (e.g., stock prices, futures prices, etc.), and any other data being received and its sequence information (e.g., the order in which it is being received) recorded by the log engine.

[0010] In one embodiment, a log engine for recording data operations in a file system is disclosed. The log engine includes a user interface module configured to receive a plurality of data operation logs, the plurality of data operation logs respectively recording data operations and sequence IDs corresponding to the order of the data operations being executed; and a shared log manager configured to provide a lock-free in-queue, the in-queue configured to integrate the plurality of data operation logs, and exchange the plurality of data operation logs to an out-queue, wherein the plurality of data operation logs are sorted based on the sequence ID and stored in the shared log.

[0011] In one embodiment, the journaling engine includes a key-value file system coordinator configured to maintain a shared journal within a predefined file size allocation for updating the shared journal.

[0012] In one embodiment, when saving the shared log in persistent storage, the key-value file system coordinator is configured to omit metadata about the file size of the shared log.

[0013] In one embodiment, the shared log manager is configured to request an extension of a memory allocation for storing the shared log when the shared log has a file size that exceeds a predefined file size allocation.

[0014] In one embodiment, the shared log includes multiple data operation logs in an order based on a sequence ID and an order of extended memory allocation so that tail delays from random writes are avoided.

[0015] In one embodiment, the logging engine includes a metadata manager configured to persist a log file of the metadata log when the log file exceeds a predefined file size.

[0016] In one embodiment, the logging engine includes a garbage collection module of the logging engine configured to chronologically validate entries in the shared log.

[0017] In one embodiment, the journaling engine includes a recovery manager configured to read a checkpoint file for performing a fuzzy checkpoint to recover from a system failure.

[0018] In one embodiment, the logging engine includes a background thread configured to remove the metadata log to persistent storage when the metadata log exceeds a predefined file size.

[0019] In one embodiment, the shared log manager is configured to perform cyclic redundancy checking for error detection in the log files.

[0020] In one embodiment, a method for recording data operations in a file system is disclosed. The method includes receiving a plurality of data operation logs, each of which records data operations and a sequence ID corresponding to the order of the data operations being executed; integrating the plurality of data operation logs in a lock-free in-queue; and exchanging the plurality of data operation logs to an out-queue, wherein the plurality of data operation logs are sorted based on the sequence ID and stored in a shared log.

[0021] In one embodiment, the method includes maintaining the shared log within a predefined file size allocation for updating the shared log when metadata regarding the file size of the shared log is omitted.

[0022] In one embodiment, the method includes requesting an extension of a memory allocation for storing the shared log when the shared log has a file size that exceeds a predefined file size allocation.

[0023] In one embodiment, the method includes making a log file of the metadata log persistent when the log file exceeds a predefined file size.

[0024] In one embodiment, the method includes chronologically validating entries in the shared log for garbage collection.

[0025] In one embodiment, the method includes reading a checkpoint file for performing a fuzzy checkpoint to recover from a system failure.

[0026] In one embodiment, a computer-readable medium includes instructions that, when executed by a processor, instruct the processor to receive a plurality of data operation logs, each of which records data operations and a sequence ID corresponding to the order in which the data operations are being executed; provide a lock-free in-queue configured to integrate the plurality of data operation logs; and exchange the plurality of data operation logs to an out-queue, wherein the plurality of data operation logs are sorted based on the sequence ID and stored in a shared log.

[0027] In one embodiment, the computer readable medium further comprises instructions that, when executed by the processor, maintain the shared log within a predefined file size allocation for updating the shared log when metadata regarding the file size of the shared log is omitted.

[0028] In one embodiment, the computer readable medium further comprises instructions that, when executed by the processor, request expansion of memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.

[0029] In one embodiment, the computer-readable medium further comprises instructions that, when executed by the processor, cause a log file of the metadata log to be persistent when the log file exceeds a predefined file size.

[0030] In one embodiment, the computer readable medium further comprises instructions that, when executed by the processor, chronologically validate entries in the shared log for garbage collection.

[0031] In one embodiment, the computer readable medium further comprises instructions that, when executed by the processor, read a checkpoint file for performing a fuzzy checkpoint to recover from a system failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a key-value store and a file system according to one embodiment is shown.

[0033] Figure 2 Modules of a key-value store and a file system are shown according to one embodiment.

[0034] Figure 3 A key-value store and file system that handles multiple I / O streams is shown according to one embodiment.

[0035] Figure 4 The architecture of a log engine for recording log data according to one embodiment is shown.

[0036] Figure 5 A design of a log engine for sharing logs according to one embodiment is shown.

[0037] Figure 6 Data management for a shared log according to one embodiment is shown.

[0038] Figure 7 Metadata management according to one embodiment is shown.

[0039] Figure 8 Persisting metadata to disk according to one embodiment is shown.

[0040] Fig. 9 is based on Figure 4 Schematic diagram of a checkpoint and compression manager of one embodiment.

[0041] Fig.10 A design of a recovery manager according to one embodiment is shown. DETAILED DESCRIPTION

[0042] The present disclosure relates to key-value storage and file systems. In particular, the present disclosure relates to a log engine for managing log data of key-value storage and file systems.

[0043] In the following detailed description, reference is made to the accompanying drawings that form a part of the description. In the accompanying drawings, similar symbols generally identify similar components unless the context dictates otherwise. In addition, unless otherwise indicated, the description of consecutive drawings may refer to features from any previous drawings to provide a clearer context and substantive explanation of the current exemplary embodiment. Nevertheless, the exemplary embodiments described in the detailed description, drawings, and claims are not intended to be restrictive. Other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure, as generally described and narrated herein and as shown in the accompanying drawings, may be arranged, replaced, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0044] In addition, parts of the present disclosure may be described herein in terms of functional block components and various processing steps. It should be understood that such functional blocks may be implemented by any number of firmware, software, and / or hardware components configured to perform the specified functions.

[0045] Figure 1 A schematic diagram of a key-value store and file system according to an embodiment is shown. The key-value store and file system 100 includes a key-value system 102. The key-value system 102 includes a log engine 104, a hash engine 106, a sort engine 108, and a garbage collection manager 110. The key-value store and file system 100 also includes a collaboration layer 112 and a file system 114. The key-value store and file system 100 can interact with a kernel space 116, which includes one or more disks 118. The key-value store and file system 100 can interact with an application 120.

[0046] The key-value and file system 100 can be used for storage in cloud applications, for example, to provide data persistence required by cloud services. The key-value system 102 can be configured to provide key-value storage, for example, as part of a storage backend for a cloud service. Non-limiting examples of cloud services that use the key-value system 102 include shopping, social media, metadata management, etc. The file system 114 can be a dedicated user-level append-only file system that is configured to provide storage dedicated to facilitating the operation of the key-value system 102.

[0047] The log engine 104 is configured to allow concurrent writing of multiple log files, thereby reducing the number of compression and garbage collection operations. The logs written by the log engine 104 can be configured so that the handling of the logs does not require strong ordering. The log engine 104 is configured to improve throughput performance issues in log writing and increase recovery speed by reducing the synchronous write overhead of the log from multiple input / output (I / O) operations to a single I / O operation, using lock-free queues to aggregate writes to control latency and improve throughput, and / or providing asynchronous interfaces to enhance the threading model. In the case where the key-value system 102 and the file system 114 are integrated and collaborated, the log engine 104 can be used to store a write-ahead log (WAL) with a predefined structure that has a defined actual file size. The defined file size for the WAL can then result in fewer I / O operations being required, thereby enhancing performance while mitigating potential trade-offs regarding data consistency. The operation of the log engine 104 is Figures 4 to 8 is further detailed in and described below.

[0048] The hash engine 106 is configured to handle point queries within the key-value system 102. In particular, the hash engine 106 is configured to reduce tail latency in point queries. The hash engine 106 includes a separation of data components and index components, and maintains indexes in cache memory, such as by compressing the index and / or caching partial data. The partial data can be selected using, for example, a least recently used strategy.

[0049] The sorting engine 108 is configured to perform range scan operations while reducing write amplification factors and / or read / write latency associated with such operations. The sorting engine 108 is configured to use a partitioned log structure merge (LSM) tree. Classification of I / O streams and scheduling of tasks may also be performed by the sorting engine 108.

[0050] The garbage collection manager 110 is configured to perform garbage collection and / or compression operations in the key-value and file system 100. The garbage collection manager 110 can be configured to reduce unnecessary data movement during garbage collection and / or compression operations in the key-value and file system 100. The garbage collection manager 110 can perform garbage collection and / or compression operations based on the perception of application-side data deletions such as page expiration. The garbage collection and compression performed by the garbage collection manager 110 can be configured to arrange data to support other modules such as the sorting engine 108. The garbage collection manager 110 can coordinate the preservation of data during garbage collection and / or compression operations.

[0051] The collaboration layer 112 is configured to facilitate collaboration between the key-value system 102 and the file system 114. The collaboration layer 112 can also facilitate efficient compression and / or garbage collection operations in the key-value system 102 based on the collaboration between the key-value system 102 and the file system 114. Collaboration can reduce write amplification problems caused by compression and / or garbage collection operations. In one embodiment, the collaboration layer 112 can expose region usage information from the key-value system 102 to the file system 114.

[0052] The file system 114 is configured to separate data from the log and use a log-structured append-only write as the write model. In one embodiment, the file system may also provide pre-allocated data space, where synchronous writes occur only for data persistence, and in one embodiment, metadata persistence is not required. In one embodiment, data persistence and global log persistence for different files may be performed separately. These aspects of the file system may allow the file system to avoid some metadata persistence operations, such as those caused by a single data write persistence operation.

[0053] The file system 114 is configured to support general files and instant files. General files and instant files can be written sequentially, and both can be read sequentially or randomly. General files can be optimized to achieve continuous low latency in continuous or random reads. General files can be used to write data in batches without flushing the data to disk after each write, such as SST files. Storage space is allocated in large units, where a non-limiting example of the unit size is 1MB each. Large allocation units can reduce the metadata size of general files so that during normal file system operations, the metadata of all general files can be saved in memory. By saving metadata in memory, read operations on general files do not require further I / O to access metadata regardless of the read offset. This can reduce the read tail latency of general files. Instant files can be optimized for fast incremental synchronous writes while having good sequential and random read performance near the tail. Instant files can be used to write data that needs to be frequently refreshed to disk for instant persistence, such as pre-write log files of key-value systems. Each individually written data and metadata can be bundled together for instant files. The bundled data and metadata can be written to a journal file shared by all instant files. Bundling data and writing to a journal file can improve the speed of incremental writes and sync operations. This approach is structured to support sequential reads, but can make tradeoffs with respect to random reads. Since live files are expected to be mostly read sequentially, with random reads mostly concentrated near the tail, the most recently written data for each live file being actively written can be cached to improve read performance.

[0054] The file system 114 may include a user space I / O scheduler to assign I / O priorities to different I / O types. The I / O scheduler marks the foreground I / O as high priority, while the background I / O will be marked as low priority. In addition, the key-value system 102 may include a scheduler to schedule its background tasks to ensure that each I / O issued by the upper layer application has consistent I / O amplification. Through this common design of I / O scheduling in both the key-value system 102 and the file system 114, since both I / O amplification and I / O latency are consistent, tail latency can remain stable and low. In addition, reading common files from the file system does not require I / O for metadata, and using a large space for common files ensures that most read operations require a single I / O.

[0055] Kernel space 116 may contain disks 118. Disks 118 may include one or more storage media, such as solid-state drives (SSDs). In an embodiment, at least some of disks 118 are zoned storage (ZNS) SSDs.

[0056] Application 120 is any suitable application that utilizes key-value store and file system 100, such as online shopping, social media, metadata management applications, etc. Application 120 may interface with key-value and file system 100 via any suitable application programming interface (API). In one embodiment, the API may be specific to a particular type of file, such as having properties of a file as a generic file or an instant file determined by the API through which the file has been received.

[0057] Figure 2 Modules of a key-value store and file system according to an embodiment are shown. The key-value store and file system 200 includes a logging engine 202, a hashing engine 204, a sorting engine 206, a garbage collection manager 208, and a file system 210.

[0058] The log engine 202 is configured to allow concurrent writing of multiple log files, thereby reducing the number of compression and garbage collection operations. The logs written by the log engine 202 can be configured so that processing of the logs does not require strong ordering. The log engine 202 is configured to improve throughput performance issues in log writing and increase recovery speed by reducing the synchronous writing overhead of the log from multiple input / output (I / O) operations to a single I / O operation, using lock-free queues to aggregate writes to control latency and improve throughput, and / or providing asynchronous interfaces to enhance the threading model. The operation of the log engine 202 is Figures 4 to 8 Detailed description is given below.

[0059] The hash engine 204 is configured to handle point queries within the key-value and file storage system 200. In particular, the hash engine 204 is configured to reduce tail latency in point queries. The hash engine 204 includes a separation of data components and index components, and maintains the index in a cache memory, such as by compressing the index and / or caching partial data. The partial data can be selected using, for example, a least recently used strategy.

[0060] The sorting engine 206 is configured to perform range scan operations while reducing write amplification factors and / or read / write latency associated with such operations. The sorting engine 206 is configured to use a partitioned log structured merge (LSM) tree. Classification of I / O streams and scheduling of tasks may also be performed by the sorting engine 206.

[0061] The garbage collection manager 208 is a proactive garbage collection manager for the key-value store and file system 200. The garbage collection manager 208 can be configured to reduce unnecessary data movement during garbage collection and / or compaction operations in the key-value and file system 200. The garbage collection manager 208 can perform garbage collection and / or compaction operations based on awareness of application-side data deletions, such as page expiration. The garbage collection and compaction performed by the garbage collection manager 208 can be configured to arrange data to support other modules, such as the sorting engine 206. The garbage collection manager 208 can coordinate the preservation of data during garbage collection and / or compaction operations.

[0062] The file system 210 can be an append-only file system for key-value storage. The file system 210 can be configured to separate data from the log and use log-structured append-only writes as the write model. In one embodiment, the file system 210 can also provide pre-allocated data space, where synchronous writes occur only for the persistence of data, and in one embodiment, metadata does not need to be persisted. In one embodiment, data persistence and global log persistence for different files can be performed separately. These aspects of the file system can allow the file system to avoid some metadata persistence operations, such as those caused by a single data write persistence operation.

[0063] Figure 3 A key-value store and file system that handles multiple I / O streams is shown according to one embodiment. Figure 3 Shows Figure 1 and Figure 2 The use of the modules shown and also described herein. Figure 3In the illustrated embodiment, the key-value store and file system is interfaced with a page server of a MySQL-compatible OTLP database. The interface with a page server of a MySQL-compatible OTLP database is a non-limiting example of a suitable application for interfacing with a key-value store and file system according to one embodiment. Figure 3 As shown, the database page server utilizes a key-value store and a file system to store each of the base page, MVPage, incremental log, and redo log. The key-value and file system can use a log engine to index and persist the redo log from the database page server. These operations can have read and write amplification, each amplification of 1. MVPage can be processed using a hash engine. Incremental logs and metadata can each be processed using a sorting engine.

[0064] Figure 4 The architecture of a log engine 400 for recording log data according to one embodiment is shown. The log engine 400 is configured to receive user input and record the order of operation of the input. It should be understood that the log engine 400 can be configured to record any time series data, such as scientific or experimental data over time (e.g., temperature, pressure, concentration, driving log, etc.), financial or economic data over time (e.g., stock prices, futures prices, etc.), and any other data being received and its sequence information (e.g., the order in which it is being received) recorded by the engine. In one embodiment, the log engine 400 can be as described above and in Figure 1 or Figure 2 The logging engine 201 or logging engine 202 discussed in.

[0065] The logging engine 400 includes a user profile manager 410 , internal components 415 , and a key-value file system (KV-FS) coordinator 490 .

[0066] The user profile manager 410 provides an interface between the internal component 415 and the user client. The user profile manager 410 interfaces between the internal component 415 and the user client to provide synchronous and asynchronous communications through an input / output (I / O) application programming interface (API). The synchronous API 412 can be configured to communicate in real time between the user client and the internal component 415, receive instructions and / or record input from the user client. The asynchronous API 414 can be configured to queue or batch I / O instructions and process I / O once resources for processing I / O become available or are allocated to batched I / O.

[0067] The synchronous or asynchronous API includes one or more functional APIs that perform data processing functions, for example, to process instructions or I / O related to a user client. The functional API includes a log file organizer, a log file editor, and a log file iterator. By providing a functional API, and / or limiting I / O functions outside the provided functional API, the API ensures that log files are managed efficiently and reliably using a predetermined pattern.

[0068] A log file manager or log file factory provides methods to create, open, and delete log files through, for example, a LogFileFactory object.

[0069] The create function includes, upon receiving a request or instruction from a user client, creating a log file with a specified name. In one embodiment, the create function includes returning a status indicator indicating whether the create function was successful.

[0070] The open function includes, upon receiving a request or instruction from a user client, opening a log file. The log file can be identified by the log file name included in the request or instruction. The open method includes, upon opening the log file, returning a status indicator indicating whether the open function is successful.

[0071] The delete function includes deleting the log file upon receiving a request or instruction from the user client. The log file can be identified by the log file name provided in the request or instruction. If the deletion is successful, the delete function returns a status indicator of the completion of the deletion (e.g., by including an indicator of a return message absl::OkStatus()).

[0072] If the create, open and / or delete functions cannot be completed, the corresponding function returns an error indicating that the log file does not exist (e.g., by including an indicator with a return message of "NotFound") or an error status indicating the cause of the failure.

[0073] The log file editor facilitates the user to open, modify, and close log files based on requests or instructions received from the user client. The log file editor or log file editor API provides users with access to interact with log files and ensures proper appending, deletion, and retrieval of log data. The log file editor includes put functions, trim functions, and truncation functions.

[0074] The placement function appends multiple bytes of log data to the log file at the end of the log file. The placement function may request or receive a sequence ID (e.g., a sequence number or "sequence number (seqnum)") in a request or instruction from a user client. The sequence ID from the user client increases monotonically to the current maximum sequence ID to ensure the proper ordering of log files recorded over time. In one embodiment, the sequence ID is a unique number that sorts the log data stored on the disk in order. It should be understood that the total order of the log data can be organized sequentially on the disk so that delays from random writes are eliminated. If the sequence ID provided by the user client is equal to or less than the current maximum sequence ID, the placement function returns an error indicating a sequence ID error. When successfully appended, the placement function will return an indicator indicating that the append operation is complete (e.g., by returning a message including "absl::OkStatus()"). If the append operation cannot be completed, the placement function returns an error code corresponding to the error.

[0075] The prune function deletes log records with a sequence ID smaller than the provided sequence ID. The truncate function deletes log records with a sequence ID larger than the provided sequence ID. Log records with the provided sequence ID can be deleted by the prune function, the truncate function, etc.

[0076] The log file editor or LogFile includes a size function that returns the size of the log data available to the user, a log count function (such as a NumRecords function) that returns the total number of records in the log file, a close function that closes the log file thereby releasing occupied resources and providing proper resource management.

[0077] The log file iterator facilitates users to read log files sequentially, which is convenient and efficient for users to traverse and read log files in a controlled manner. The log file iterator includes one or more functions.

[0078] The iterator function creates an iterator for a log file (e.g., the log file currently being viewed by a user) and returns an integrator pointer if successful. The seek function requests or receives a sequence ID from a user client and positions the iterator to the first record whose sequence ID is greater than or equal to the sequence ID. The validate function checks whether the record pointed to by the iterator is valid. The next function moves the iterator to the next record in the log file. The sequence function returns the sequence ID of the log record. The size function returns the size of the data in the log record. The status function retrieves the current state of the iterator (e.g., whether the function succeeded or failed). The read function reads data from the current log file pointed to by the iterator. The read function can obtain the internal offset within the log file and the number of bytes to be read, and returns an indicator indicating that the read operation was successful or an error code corresponding to an error.

[0079] Internal components 415 include a shared log manager 420, a metadata manager 430, a background task manager 440, and a recovery manager 450, which are disposed between the file system 470 and the user interface manager 410 to facilitate I / O requests originating from user clients.

[0080] The shared log manager 420 consolidates data processing requests (e.g., append, create, delete, trim and / or truncate functions, etc.). The shared log manager 420 creates and manages consolidated log data 417, which includes metadata 424, user data 426, and a corresponding shared log of a checkpoint (CPT) file 422. In one embodiment, the shared log manager 420 uses "in queues" and "out queues" to efficiently handle data by consolidating processing requests from user clients. In one embodiment, when rolling, the shared log 426 for user data is encoded and persisted in the file system as a data log 486. Reference will be made below to Figure 5 and Figure 6 A shared log manager 420 is also described.

[0081] The metadata manager 430 creates, prunes, and saves metadata 419 for the log engine 400. The metadata 419 may include a namespace 432 that maintains a mapping of file names to file identifiers. A file map 434 tracks the mapping of file identifiers to LSM meta-instances. The metadata manager 430 moves the metadata 419 to a persistent storage device after a threshold event. Figure 7 and Figure 8 Metadata manager 430 is described.

[0082] Background task manager 440 organizes the execution of background functions, thereby providing background functions for maintaining engine 400. In the example shown, the background module uses one or more background threads to provide data processing. Background task manager 440 includes garbage collection manager 442, checkpoint and compression manager 444 and L0Dump manager 446.

[0083] The garbage collection manager 442 identifies and deletes dirty bytes, which are data that becomes invalid due to, for example, pruning, deleting, and truncating functions in the shared log manager 420, for effectively managing and removing unnecessary data and optimizing system performance and storage utilization.

[0084] In one embodiment, when one or more operations that may create dirty bytes are performed, the garbage collection manager 442 starts with garbage data marking. For example, garbage data marking can be triggered by executing a trim or delete function in the shared log manager 420. The garbage collection manager 442 then proceeds to garbage data calculation.

[0085] At garbage data calculation time, the garbage collection manager 442 adds the size of the newly created dirty bytes to the garbage data statistics stored in volatile or persistent memory. The garbage data statistics include the actual size or proxy size of the accumulated garbage data.

[0086] In one embodiment, where the user data is fully indexed and / or directly indexed in a memory (e.g., in a volatile memory or L0), the newly generated dirty bytes may include the user data and metadata. The size of the newly generated dirty bytes may be the sum of the user data and metadata in L0. In another embodiment, where the user data is indirectly indexed, the dirty bytes may include an L0 portion and an L1 portion. The L0 portion may be data in a volatile memory. The L0 portion dirty bytes include the user data and metadata in L0. The L1 portion may be data in a persistent storage device. In one embodiment, the size of the user data or dirty bytes may be estimated as the average size of the user data in the persistent storage device, thereby avoiding reading indirect indexes for size calculations during the completion of pruning or deletion functions of front-end functions, etc. Using the average size of the user data as a proxy for the actual size of a group of user data in the L1 portion reduces computational overhead.

[0087] Once a threshold event triggering garbage data calculation occurs, the garbage collection manager 442 performs re-entry garbage data statistics and performs a garbage collection process.

[0088] In one embodiment, the threshold event is when the total amount of garbage data (e.g., the size of the accumulated garbage data) exceeds the capacity of one physical log, constitutes a certain percentage (e.g., 20%, 30%, 40%, etc.) of the total system data, and so on. If one or more conditions are met, the threshold event occurs. In one embodiment, the threshold event occurs when both conditions (e.g., exceeding the capacity of one physical log and exceeding 30% of the total system data) are met. It should be understood that the total system data can be the total size of user data and metadata in the memory, and if available, can be the total L1 portion of data estimated based on the average size of user data.

[0089] It should be understood that dirty bytes or invalid files are usually unmarked. When performing garbage collection, the garbage collection manager 442 usually starts by determining the validity of the data in the file. In one embodiment, the garbage collection manager 442 determines the validity of the file in chronological order (e.g., from the oldest to the newest) because older log files are more likely to no longer be used and become invalid. In one embodiment, the log files are sorted by sequence ID, such as older log files have smaller sequence IDs, and newer files have larger sequence IDs. The validity check of garbage collection is performed by a background thread. The background thread traverses and scans the oldest log file, and determines the data validity by querying the index metadata (e.g., for user data). If the corresponding user data is found, the log file is valid and is moved to a newer location. In one embodiment, the newer location can be the latest log file location. If the corresponding user data cannot be found, the oldest log file is invalid. The oldest log file is marked as dirty and / or deleted.

[0090] Then, recalculate and / or re-enter the garbage data statistics to update the garbage data statistics. In one embodiment, the garbage data is indexed in the memory, and the garbage data statistics are updated and recorded in the memory. Upon re-entry, the total amount of garbage data is recalculated based on the total system data and the verified data. The garbage collection manager 442 determines again whether the threshold event still occurs to trigger the garbage data calculation. If the threshold event still exists, the garbage collection manager 442 scans the next oldest log file that is now in the oldest position for verification. If the threshold event no longer exists, the garbage collection manager 442 ends the garbage collection process.

[0091] In one embodiment, after the log file validity is determined to be valid and moved, foreground logic (e.g., instructions from a user to delete or prune the log file) invalidates the log file and newly written log files are marked as dirty and / or deleted.

[0092] In one embodiment, during the verification of the log file, the background thread exits the garbage collection process and re-enters to continue garbage collection, comparing the index validity and storage address of the user data to determine whether the log data has been verified. In this case, the verified log file does not need to be moved again.

[0093] When the metadata in the metadata log reaches or exceeds a predetermined size for creating a new checkpoint file, the checkpoint and compression manager 444 rolls the metadata log. The new checkpoint file can then be saved in the file system (e.g., shown as cpt log 482). The checkpoint and compression manager 444 is described below with respect to Fig. 9 describe.

[0094] When the data log reaches or exceeds a predetermined size, the L0Dump manager 446 rolls the data log. The data log can be a file containing a series of log information or log data. The L0Dump manager 446 creates a new data log to continue to save new log information in the new data log. The rolling process allows the durability and reliability of the log engine 400 by persistently storing metadata and volatile memory log data (e.g., L0 information) during the data log rolling event. At least a portion of the log information is saved as data in memory (e.g., L0 information stored in volatile memory), which can be periodically refreshed to a persistent storage device (e.g., shown as data log 486) in the file system to maintain consistency and prevent data loss, for example, during a power outage, system failure, or crash.

[0095] The L0Dump manager 446 starts the roll by scanning the namespace 432. The namespace 432 maps file names to their corresponding metadata, which contains information such as file ID, offset, sequence ID, etc.

[0096] The L0Dump manager 446 then encodes the data in the memory into a persistent format file (e.g., L0Dump format) for a persistent storage device. The persistent format file is then saved in a persistent storage device (e.g., on an SSD, disk, etc.), and the metadata in the persistent format file is written to Metalog, which is a log file that records metadata and other information, for example, for recovery. The L0Dump manager 446 continues to scan, encode, and store persistent format files until all files in the namespace 432 are processed. This ensures that the metadata of the log data is recorded in the Metalog to support recovery in the event of a system failure or crash.

[0097] The KV-FS collaborator 490 is coupled between the log engine 400 and the file system 470 to synchronize the integrated log data 417 in the log engine 400 and the file system 470. The log files used in the file system and the log engine 400 are append-only files. Typically, a synchronization function (e.g., fsync or fdatasync) triggers two separate I / O operations: one for updating data and the other for updating the file size, resulting in increased input / output overhead and write amplification factor (WAF). For example, fsync ensures data consistency by synchronizing both the user data (i.e., the first I / O) and metadata (i.e., the second I / O) of a particular file. The function fdatasync only synchronizes the required data to ensure that subsequent data retrieval can be handled correctly. For append-only files being synchronized, fdatasync synchronizes both user data and metadata, triggering two I / O operations like fsync.

[0098] The KV-FS collaborator 490 synchronizes user data, but does not synchronize the file size that is usually stored in the metadata of the user data unless a new storage unit or storage extension is allocated to accommodate the new user data. In one embodiment, the file size is predefined (e.g., by WAL) and can be identified by user applications, upper layer software, etc. It should be understood that the allocation of a new extension or the next extension is a relatively infrequent event compared to the overall number of synchronizations, thereby reducing the number of system operations in most synchronization situations. Therefore, the KV-FS collaborator 490 avoids I / O operations for data size communication during synchronization, for example, by omitting the transmission of metadata about file size. In one embodiment, the metadata of user data is omitted. Compared with the synchronization of only appended files that always includes I / O for file size, the KV-FS collaborator 490 effectively reduces the total WAF by 1. This results in improved performance and enhanced data consistency, thereby improving performance and data integrity.

[0099] Figure 5 The design of a log engine for sharing logs according to one embodiment is shown. The log engine is configured to store log data in upper layer software, such as database software. In one embodiment, Figure 5 The log engine may be log engine 104, log engine 202 or Figure 4 Log engine 400.

[0100] like Figure 5 The logging engine shown includes a user interface manager (e.g., Figure 4 410), which may be implemented, for example, via a synchronous or asynchronous API (e.g., Figure 4 412 or 414) to receive one or more user requests (e.g., Op1 and Op2). The log engine may also include an internal layer configured to manage metadata, shared logs, and background tasks (e.g., Figure 4 Internal components 415). Figure 5 The logging engine shown includes a backend layer (e.g. Figure 4 440), which is configured to interface with the file system, for example, by writing logs to a physical log to make those logs persistent.

[0101] The log engine is configured to create one or more shared logs, which integrate multiple user operations into a predetermined number of I / O requests. In the example shown, the log engine includes multiple user operations into a shared log, and integrates multiple users into a single I / O request. It should be understood that the shared log can be a shared log of a checkpoint file, user data, or metadata. Non-limiting examples of such user operations include appending, creating, deleting, pruning, and truncating operations. These integrated operations can be handled by a predetermined number of I / O requests to the file system. In one embodiment, user operations can be integrated into a single shared log and processed as a single request. In one embodiment, operations can be integrated into two or more requests respectively in two or more shared logs, wherein threads process each shared log respectively. It should be understood that the number of shared logs is usually less than the number of users, so that the processing power of the processing core is fully utilized.

[0102] like Figure 5 As shown in the exemplary embodiment of FIG, the log engine receives operations Op1 and Op2 through a user interface. The received operations Op1 and Op2 enter the enqueued state of the internal layer from the queued state. The internal layer is configured to exchange the enqueue with the outqueue under special conditions (such as the availability of processing threads), at which time the outqueue can flush all previous contents of the enqueue to the file system, thereby making various requests currently in the enqueue persist in the file system through a single I / O operation of the shared log.

[0103] In one embodiment, multiple instances of a shared log can be used to manage different forms of data, such as user data, metadata, checkpoint files, and other files. User data can be log data stored by the log engine. Metadata for improving read operations can also be included in the instance of the shared log within the log engine. Metadata management can include one or both of a metadata storage module in memory using standard mappings and persistent metadata storage on the storage device memory. The persistent metadata storage can take a format similar to a log structure merge (LSM). Metadata management within the log engine is described below and described in Figure 7 Shown in.

[0104] Figure 6 FIG. 2 shows data management of a shared log according to an embodiment. Figure 6 As shown, one or more clients request three "append" operations (i.e., append(1, "hello"), append(4, "engine"), and append(3, "log")). In the example shown, sequence IDs (i.e., 1, 4, and 3) are provided in the operations, and the sequence IDs are executed from smallest to largest. The log engine receives these operations as they are queued. From the queuer, the operations are added to the shared user data log, as described above with respect to Figure 5As described, the received user operations are packaged into log records, and the background thread is waited for to flush all requests to disk, making the log updates persistent, as shown in DataLog (e.g., a log file recording the operation log), where the corresponding records each include the corresponding requested append. The order of the operations is preserved according to their sequence IDs.

[0105] Figure 7 Metadata management according to one embodiment is shown. In particular, Figure 7 The append, create, delete, and prune requests from the client are shown, as well as the operations in memory and on disk in response to the requests. Metadata management can be performed by the logging engine as described above and in Figures 4 to 6 Shown in.

[0106] Metadata management can include managing the file system's namespace and file mappings, handling storage metadata for each log file, and mechanisms for metadata persistence by writing to disk. The namespace can maintain a mapping of file names to file identifiers, while the file mapping can track the mapping of fids to persistent metadata instances, such as described above with respect to Figure 5 Each LSM meta-instance can maintain a mapping of sequence IDs to log record locations in the underlying file system (e.g., lsn:<fid,offset,lsn> ). An LSM meta-instance may include both level 0 (L0, directly indexed) and level 1 (L1, indirect indexed) portions in memory.

[0107] For an append operation, a corresponding entry is added to the metadata of the corresponding log file. In one embodiment, it is recommended that the size of each such entry be 24B or approximately 24B, thereby accounting for approximately 3% of the total data with a 1k record size. As the log data grows, the memory occupied by the metadata may increase significantly. For example, the metadata capacity of a 1T disk is close to 30GB. Therefore, it may not be feasible to store all log file metadata in memory. In one embodiment, the log engine may periodically write the log file metadata to disk to make the log file metadata persistent. In one embodiment, the metadata in L0 is encoded by a L0Dump operation and persisted in the meta log on L1, such as Figure 8 Making log file metadata persistent can be an access pattern involving frequent writes and infrequent reads, where the probability of reading recently written data is high. To address this issue, the log engine can maintain the latest metadata entries in memory, write older entries to disk, and employ an indirect mapping table in memory to point to the metadata entries on disk.

[0108] Figure 8Persisting metadata to disk according to one embodiment is shown. Persisting metadata may include log file creation and / or deletion. Persisting metadata may include modifications to namespace and file mappings, pruning log files, requiring actions to be performed on LSM metadata for efficient metadata management, and / or a "L0Dump" operation that involves persisting an L0 map by encoding the L0 map to disk and saving the on-disk location of the L0 map in L1.

[0109] Fig. 9 is based on Figure 4 Schematic diagram of a checkpoint and compression manager 444 of an embodiment of . The checkpoint and compression manager 444 is triggered when the metadata log file in the memory undergoes a rollover, for example, when the metadata log exceeds a predetermined file size.

[0110] The checkpoint and compression manager 444 begins rolling by scanning the namespaces to obtain metadata saved in volatile memory (e.g., in L0) at the first namespace. The L0Dump function encodes the L0 information into a persistent data format to be combined with the L1 metadata. The L1 metadata is already saved in a persistent data format. The manager 444 then performs compression to combine the L1 metadata, the encoded L0 metadata, and the CPT file (e.g., a previous checkpoint file). This compression process generates formatted metadata that is configured to be permanently stored in the CPT file. The compressed metadata is written to the new CPT file. The L1 in memory is also updated to merge and make the L0 metadata of the first namespace persistent. The manager 444 then advances to the next namespace and iterates until all metadata in L0 has been processed and written to the new CPT file.

[0111] In one embodiment, manager 444 uses fuzzy checkpoints to reduce thread blocking during the metadata rolling process. Manager 444 locks a small portion of the LSM metadata for each file for a short period of time, thereby avoiding activating the global lock of the entire log engine. By using fuzzy checkpoints, interference with user threads is reduced, thereby ensuring smoother and more efficient management of the metadata and name space of the log engine.

[0112] Fig.10 The design of a recovery manager according to one embodiment is shown. In an embodiment, the recovery manager can be based on Figure 4 The recovery manager 450 of an embodiment.

[0113] like Fig.10As shown, the recovery manager recovers the data log after a crash or failure. In an embodiment, the recovery manager includes a ReadCheckpoint function, a ScanMetalog function, a ScanDatalog function, and a recoverL0 function.

[0114] The ReadCheckpoint function reads the checkpoint file to rebuild data structures such as the namespace (mapping file names to file IDs), the file map (mapping file IDs to LSM meta instances), and the L1 map for each log file. The checkpoint provides a starting point for recovery. For example, the recovery manager begins the recovery process by reading the checkpoint file and using the data stored therein to rebuild the namespace, file map, and L1 map for each log file (e.g., Metalog, Datalog, etc.).

[0115] The ScanMetalog function instructs the recovery manager to scan the Metalog file pointed to by the checkpoint. The ScanMetalog function updates the namespace, file mapping, and L1 mapping based on the operations recorded in the Metalog. In order to optimize recovery time, a "fuzzy checkpoint" method is used to skip redundant operations. For example, if a delete operation is encountered, the recovery manager checks whether the file name exists in the namespace. If not, the delete operation has been successfully committed and does not need to be redone.

[0116] For example, the Metalog file referenced by the checkpoint is scanned to update the namespace, file mapping, and L1 mapping. Then, in order to reduce interference to the user, the recovery manager locks the specific file being processed and keeps other files unlocked, so that the user can modify other files while the recovery manager is performing the checkpoint operation, thereby continuing the checkpoint operation. The recovery manager can adopt a fuzzy checkpoint method, where the time period of the checkpoint overlaps with the time period of the latest metalog, so that certain operations can be skipped.

[0117] In one embodiment, when the recovery manager encounters a delete operation, the recovery manager checks or determines whether the file name exists in the name space. If the file name does not exist, the delete operation has been successfully performed and the delete operation is skipped.

[0118] In one embodiment, when the recovery manager encounters a create operation, the recovery manager checks or determines whether the file name exists in the name space. If the file exists, the create operation has been successfully performed and the create operation is skipped.

[0119] In one embodiment, when the recovery manager encounters a prune operation, the recovery manager checks the maximum sequence ID value stored in the checkpoint file. The maximum sequence ID value is compared with the maximum sequence ID value in the data log in L0, and if the maximum sequence ID reflects the maximum sequence ID provided after the prune operation, then the prune operation has been successfully performed and the prune operation is skipped.

[0120] In one embodiment, the recovery manager suspends the L0Dump function during the checkpoint process. The L1 map is not updated because the single background thread shared between the recovery manager and the L0Dump manager 446 is occupied by the recovery manager. Once the recovery manager completes scanning the Metalog, the L1 map is restored.

[0121] The ScanDataLog function then scans the data log to replay the appended entries. Updates in the appended entries are replayed based on the sequence ID stored during the L0Dump operation. For example, the recovery manager 450 scans the Datalog. Using the data sequence ID from L0Dump, an evaluation is performed to determine whether the appended entries need to be replayed. Only updates within the appended entries are replayed.

[0122] Then, the recoverL0 function includes the recovery manager recovering the L0 entries of each LSM metadata after scanning the Datalog, thereby ensuring that the data structure is consistent. For example, after the data log scanning and recovery process is over, the L0 entries of each LSM metadata are recovered. The recovery process has reached its apex, and the log engine is operational and able to accept user read and write requests. At this point, the recovery process is complete, and the log engine is recovered and ready to accept user read and write requests. The recovery manager has ensured that the system's metadata and data structures are in a consistent state, thereby providing reliability and durability for the log engine.

[0123] aspect:

[0124] It should be understood that aspects 1 to 8, 21 and 22 may be combined with any one of aspects 9 to 15 and / or aspects 16 to 20.

[0125] Aspect 1. A log engine for recording data operations in a file system, the log engine comprising: a user interface module, configured to receive multiple data operation logs, the multiple data operation logs respectively recording data operations and sequence IDs corresponding to the order in which the data operations are being executed; and a shared log manager, configured to provide an in-queue configured to integrate the multiple data operation logs, and to exchange the multiple data operation logs to an out-queue, wherein the multiple data operation logs are sorted based on the sequence IDs and stored in a shared log.

[0126] Aspect 2. The log engine according to Aspect 1 further includes a key-value file system coordinator configured to keep the shared log within a predefined file size allocation for updating the shared log.

[0127] Aspect 3. The log engine according to Aspect 1 or 2 further includes, when saving the shared log in persistent storage, the key-value file system collaborator is configured to omit metadata about the file size of the shared log.

[0128] Aspect 4. The log engine according to any one of aspects 1 to 3, wherein the shared log manager is configured to request an extension of a memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.

[0129] Aspect 5. A log engine according to any one of aspects 1 to 4, wherein the shared log includes the plurality of data operation logs in a sequential order based on the sequence ID and the extended memory allocation, so that tail delays from random writes are avoided.

[0130] Aspect 6. The log engine according to any one of Aspects 1 to 5 further includes a metadata manager, wherein the metadata manager is configured to make the log file of the metadata log persistent when the log file exceeds a predefined file size.

[0131] Aspect 7. The log engine according to any one of Aspects 1 to 6 further comprises a garbage collection module of the log engine, wherein the garbage collection module is configured to verify entries in the shared log in chronological order.

[0132] Aspect 8. The journal engine according to any one of Aspects 1 to 7 further includes a recovery manager configured to read a checkpoint file for performing a fuzzy checkpoint to recover from a system failure.

[0133] Aspect 9. A method for recording data operations in a file system, the method comprising: receiving multiple data operation logs, the multiple data operation logs respectively recording data operations and sequence IDs corresponding to the order of the data operations being executed; integrating the multiple data operation logs in a lock-free in-queue; and exchanging the multiple data operation logs to an out-queue, wherein the multiple data operation logs are sorted based on the sequence ID and saved in a shared log.

[0134] Aspect 10. The method according to aspect 9 further comprises maintaining the shared log in a predefined file size allocation for updating the shared log when metadata about the file size of the shared log is omitted.

[0135] Aspect 11. The method according to aspect 9 or 10, further comprising requesting an extension of a memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.

[0136] Aspect 12. The method according to any one of Aspects 9 to 11 further comprises, when a log file of the metadata log exceeds a predefined file size, using a background thread to make the log file persistent.

[0137] Aspect 13. The method according to any one of aspects 9 to 12 further comprises verifying entries in the shared log in chronological order for garbage collection.

[0138] Aspect 14. The method according to any one of Aspects 9 to 13 further comprises reading a checkpoint file for performing a fuzzy checkpoint to recover from a system failure.

[0139] Aspect 15. A computer-readable medium comprising instructions, which, when executed by a processor, instruct the processor to receive a plurality of data operation logs, wherein the plurality of data operation logs respectively record data operations and sequence IDs corresponding to the order in which the data operations are being executed; provide a lock-free in-queue configured to integrate the plurality of data operation logs; and exchange the plurality of data operation logs to an out-queue, wherein the plurality of data operation logs are sorted based on the sequence IDs and stored in a shared log.

[0140] Aspect 16. The computer-readable medium according to aspect 15 further comprises instructions which, when executed by the processor, instruct the processor to maintain the shared log in a predefined file size allocation for updating the shared record when metadata about the file size of the shared record is omitted.

[0141] Aspect 17. The computer-readable medium according to aspect 15 or 16 further comprises instructions which, when executed by the processor, instruct the processor to request an extension of memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.

[0142] Aspect 18. The computer-readable medium according to any one of Aspects 15 to 17 further comprises instructions which, when executed by the processor, instruct the processor to use a background thread to make the log file of the metadata log persistent when the log file exceeds a predefined file size.

[0143] Aspect 19. The computer-readable medium according to any one of aspects 15 to 18, further comprising instructions, which when executed by the processor, instruct the processor to verify entries in the shared log in chronological order for garbage collection.

[0144] Aspect 20. The computer-readable medium according to any one of aspects 15 to 19 further comprises instructions which, when executed by the processor, instruct the processor to read a checkpoint file for performing a fuzzy checkpoint to recover from a system failure.

[0145] Aspect 21. The log engine according to any one of Aspects 1 to 8 further comprises a background thread configured to remove the metadata log to a persistent storage when the metadata log exceeds a predefined file size.

[0146] Aspect 22. The log engine according to any one of aspects 1 to 8 or 21, wherein the shared log manager is configured to perform a cyclic redundancy check on error detection in the log file.

[0147] The examples disclosed in this application are to be considered in all respects as illustrative and non-restrictive. The scope of the invention is indicated by the appended claims rather than by the foregoing description; and all changes coming within the meaning and equivalency range of the claims are intended to be embraced therein.

Claims

1. A log engine for recording data operations in a file system, the log engine comprising: A user interface module configured to receive a plurality of data operation logs, the plurality of data operation logs respectively recording data operations and sequence IDs corresponding to the order in which the data operations are being executed; as well as The shared log manager, implemented as a lock-free queue, is configured as: providing an inbound queue configured to integrate the plurality of data operation logs, and exchanging the plurality of data operation logs to a dequeue, wherein the plurality of data operation logs are sorted based on the sequence ID and stored in a shared log, wherein The shared log includes the plurality of data operation logs in an order based on the sequence ID and an order of extended memory allocation.

2. The log engine according to claim 1, further comprising: a key-value file system coordinator configured to maintain the shared log within a predefined file size allocation for updating the shared log; as well as A shared log manager is configured to request an extension of a memory allocation for storing the shared log when the shared log having a file size exceeds a predefined file size allocation.

3. The log engine according to claim 1, further comprising: When saving the shared log in the persistent storage, the key-value file system coordinator is configured to omit metadata about the file size of the shared log.

4. The log engine according to claim 1, wherein The shared log manager is configured to perform a cyclic redundancy check for error detection in the log file.

5. The log engine according to claim 1, further comprising: A background thread is configured to move a log file of the metadata log file to a persistent storage device when the metadata log file exceeds a predefined file size.

6. The log engine according to claim 1, further comprising: A metadata manager is configured to move the metadata log file to a persistent storage device.

7. The log engine according to claim 1, further comprising: A garbage collection module of the log engine, wherein the garbage collection module is configured to verify entries in the shared log in chronological order.

8. The log engine according to claim 1, further comprising: A recovery manager is configured to read a checkpoint file for performing a fuzzy checkpoint to recover from a system failure.

9. A method for recording data operations in a file system, the method comprising: receiving a plurality of data operation logs, the plurality of data operation logs respectively recording data operations and sequence IDs corresponding to the order in which the data operations are being executed; Integrate the plurality of data operation logs into a lock-free queue; as well as The plurality of data operation logs are exchanged to a dequeue, wherein the plurality of data operation logs are sorted based on the sequence ID and saved in a shared log, the shared log including the plurality of data operation logs in an order based on the sequence ID and an order of extended memory allocation.

10. The method according to claim 9, further comprising: maintaining the shared log in a predefined file size allocation for updating the shared log when metadata regarding the file size of the shared log is omitted; and When the shared log having a file size exceeds a predefined file size allocation, an extension of a memory allocation for storing the shared log is requested.

11. The method according to claim 9, further comprising: Cyclic redundancy checking for error detection in log files.

12. The method according to claim 9, further comprising: When a log file of the metadata log exceeds a predefined file size, the log file is made persistent by a background thread.

13. The method according to claim 9, further comprising: Entries in the shared log are validated chronologically for garbage collection.

14. The method according to claim 9, further comprising: Reads a checkpoint file used to perform fuzzy checkpoints to recover from system failures.

15. A computer readable medium comprising instructions which, when executed by a processor, instruct the processor to: receiving a plurality of data operation logs, the plurality of data operation logs respectively recording data operations and sequence IDs corresponding to the order in which the data operations are being executed; providing a lock-free queue configured to integrate the plurality of data operation logs; as well as The plurality of data operation logs are exchanged to a dequeue, wherein the plurality of data operation logs are sorted based on the sequence ID and saved in a shared log, the shared log including the plurality of data operation logs in an order based on the sequence ID and an order of extended memory allocation.

Images