Log storage and fast recovery method and device for virtual machine file system

Abstract

The application discloses a kind of log storage and fast recovery method and device for virtual machine file system, belong to computer technology field.The application includes: obtaining the physical alignment parameter of host side in mounting stage, and dividing log buffer, when metadata page is modified, comparing dirty page with original mirror image page, determine offset and data length, extract data to generate variation descriptor;Multiple time series adjacent variation descriptors are aggregated into log frame and compressed, and non-compressed index block is inserted between adjacent compressed log frames and stored in log buffer;After receiving the host persistent signal, the compressed log frame is spliced into batch write request and issued to storage;When crash restarts, read index block to locate uncommitted transaction, according to the inode number recorded in non-compressed index block, incremental data is covered to original metadata block, and recovery mounting is carried out.The dirty page difference log, inode index layout and alignment batch persistence are used to realize virtual machine file system log storage and fast crash recovery.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for log storage and fast recovery of virtual machine file systems. Background Technology

[0002] Virtualized environments, where multiple virtual machines share physical storage, place high demands on I / O performance and metadata consistency. Traditional file system logging mechanisms are adapted to single-machine physical scenarios and cannot coordinate with physical storage scheduling, resulting in issues such as write amplification, I / O stuttering, and slow crash recovery.

[0003] Currently, in virtualized environments, mainstream file systems generally use a block-based circular log structure to complete metadata transaction recording and consistency assurance, which is the mainstream existing technology in the current virtualized storage field. This traditional log mechanism uses fixed disk blocks as the core read and write unit and relies on circular overwriting to achieve the recycling of log space, adapting to the storage operation logic of traditional physical machines.

[0004] However, block-level circular log structures have inherent flaws. Their coarse-grained block-level recording method generates a large amount of redundant writing, leading to severe write amplification and I / O stuttering issues, and easily causing log bloat and disk expansion delays. At the same time, system crash recovery efficiency is low and resource overhead is high, making it difficult to meet the high-performance and high-availability requirements of virtualization environments. Summary of the Invention

[0005] This invention provides a log storage and fast recovery method and apparatus for virtual machine file systems, which solves the technical problems in the prior art caused by the lack of coordinated scheduling between block-level circular logs and physical storage layers, resulting in severe metadata write amplification, frequent I / O operation pauses, and excessively long crash recovery time.

[0006] In a first aspect, embodiments of the present invention provide a method for log storage and fast recovery of a virtual machine file system, including: During the virtual machine file system mounting phase, the physical alignment parameters on the host side are obtained, a log buffer is partitioned in the guest memory, and when the virtual machine kernel triggers metadata page modification, the dirty page after metadata page modification is compared with the original image page in the guest memory to determine the starting offset and data length of metadata modification, and the data of the modification interval is extracted to generate a variational descriptor. Multiple temporally adjacent variational descriptors are aggregated into log frames and compressed to obtain compressed log frames. Uncompressed index blocks are inserted between adjacent compressed log frames to form a log storage layout stored in the log buffer. The uncompressed index blocks record the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frames. Receive the persistence signal sent by the host machine, concatenate the compressed log frames in the log buffer into a batch write request that conforms to the physical alignment parameters, and send it to the host machine for storage; When the virtual machine crashes and restarts, the uncompressed index block is read to locate the uncommitted transactions before the crash. Based on the inode number recorded in the uncompressed index block, the incremental data of the corresponding partition descriptor is overwritten to the original metadata block to restore and mount the file system.

[0007] Secondly, embodiments of the present invention also provide a log storage and fast recovery device for virtual machine file systems, comprising: The comparison and extraction module is used to obtain the physical alignment parameters on the host side during the virtual machine file system mounting stage, divide the log buffer in the guest memory, and compare the dirty page after the metadata page modification with the original image page in the guest memory when the virtual machine kernel triggers the metadata page modification, determine the starting offset and data length of the metadata modification, and extract the data of the modification interval to generate a variational descriptor. The compression insertion module is used to aggregate multiple temporally adjacent variational descriptors into a log frame and perform compression processing to obtain a compressed log frame. An uncompressed index block is inserted between adjacent compressed log frames to form a log storage layout and store it in the log buffer. The uncompressed index block records the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frame. The storage module is used to receive persistent signals from the host machine, concatenate the compressed log frames in the log buffer into batch write requests that conform to the physical alignment parameters, and send them to the host machine for storage. The location recovery module is used to read the uncompressed index block when the virtual machine crashes and restarts, locate the uncommitted transactions before the crash, and overwrite the original metadata block with the incremental data of the corresponding partition descriptor according to the inode number recorded in the uncompressed index block to perform file system recovery mounting.

[0008] Thirdly, embodiments of the present invention also provide a terminal, including: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the log storage and fast recovery method for virtual machine file systems as provided in the above embodiments.

[0009] Fourthly, embodiments of the present invention also provide a medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the log storage and fast recovery method for a virtual machine file system as provided in the above embodiments.

[0010] The log storage and fast recovery method and apparatus for virtual machine file systems provided in this invention embodiment obtains the physical alignment parameters on the host side during the virtual machine file system mounting phase, divides the log buffer in the guest memory, and compares the modified dirty page with the original image page in the guest memory when the virtual machine kernel triggers metadata page modification to determine the starting offset and data length of the metadata modification, extracts the data of the modification interval to generate variational descriptors, aggregates multiple temporally adjacent variational descriptors into log frames and performs compression processing to obtain compressed log frames, and inserts between adjacent compressed log frames. An uncompressed index block is created and stored in the log buffer to form a log storage layout. The uncompressed index block records the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frame. Upon receiving a persistence signal from the host machine, the compressed log frames in the log buffer are concatenated into a batch write request conforming to the physical alignment parameters and sent to the host machine for storage. When the virtual machine crashes and restarts, the uncompressed index block is read to locate uncommitted transactions before the crash. Based on the inode number recorded in the uncompressed index block, the incremental data corresponding to the variational descriptor is overwritten onto the original metadata block, and the file system is restored and mounted. By constructing a compressed log storage structure carrying inode indexes and combining it with host machine collaborative alignment and disk persistence mechanisms and multi-core parallel overwriting mechanisms, low-redundancy log recording of file system metadata and rapid crash recovery are achieved in a virtualized environment. Attached Figure Description

[0011] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0012] Figure 1 This is a flowchart of the log storage and fast recovery method for virtual machine file systems provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of the log storage and fast recovery method for virtual machine file systems provided in Embodiment 2 of the present invention; Figure 3 This is a structural diagram of the log storage and fast recovery device for virtual machine file systems provided in Embodiment 3 of the present invention; Figure 4 This is a structural diagram of the terminal provided in Embodiment 4 of the present invention. Detailed Implementation

[0013] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0014] Example 1 Figure 1 This is a flowchart of a log storage and fast recovery method for virtual machine file systems provided in Embodiment 1 of the present invention. This embodiment is applicable to virtual machine file system log recording and crash recovery scenarios in virtualized shared storage environments, and specifically includes the following steps: Step 110: During the virtual machine file system mounting phase, obtain the physical alignment parameters on the host side, divide the log buffer in the guest memory, and when the virtual machine kernel triggers metadata page modification, compare the dirty page after metadata page modification with the original image page in the guest memory to determine the starting offset and data length of metadata modification, and extract the data of the modification interval to generate a variational descriptor.

[0015] The virtual machine file system mounting stage is a crucial process after the virtual machine starts up, during which the file system completes initialization and establishes an interaction channel with the underlying storage. It is executed before business read / write I / O is issued, enabling storage environment detection, resource allocation, and cross-domain memory path establishment in advance, providing a basic operating environment for subsequent metadata log caching and batch data collaborative disk writing.

[0016] Pre-acquiring the host machine's physical alignment parameters and pre-allocating the log buffer ensures that subsequent log frames are assembled and written according to storage alignment requirements, avoiding latency and I / O misalignment caused by dynamic memory allocation at runtime, and enabling coordinated scheduling of the storage link between the guest machine and the host machine. Optionally, during the virtual machine file system mounting phase, virtualization link detection is first performed. The file system identifies whether the underlying storage is a virtualization backend through hypercall instructions or querying the virtual bus interface. When it is determined that the underlying storage belongs to the virtualization backend scenario, the physical alignment parameters issued by the host machine are read. For example, the physical alignment parameters of the host machine's virtual disk are commonly 512B or 4KB sector boundaries. In this embodiment, the alignment parameters obtained are 4096 bytes. Subsequently, in the guest machine's memory, a contiguous memory region is requested as a log buffer through the kernel's contiguous page allocation interface. For example, a contiguous physical memory of 64MB is allocated as the log buffer. The memory mapping between the log buffer and the host machine is established based on the virtualization input / output loop mechanism, opening up the shared memory channel between the guest machine and the host machine.

[0017] By differentially comparing dirty pages with the original mirror page, only locally modified intervals are captured, generating lightweight variational descriptors. This avoids recording metadata changes across the entire page, effectively reducing the amount of log data written and mitigating write amplification issues in virtualized storage scenarios. Optionally, when the virtual machine kernel performs a metadata update operation and triggers metadata page modification, it first quickly locates the changed metadata dirty page based on the kernel dirty page bitmap. The modified dirty page in memory is then compared segment by segment with the remaining original mirror page to accurately identify the modified area within the page and determine the starting offset and effective data length corresponding to the modified area. For example, within a standard 4KB metadata page, only the inode attribute update occurs in the offset range of 128 to 511 bytes, while the remaining 3712 bytes of data remain unchanged. Subsequently, only the binary data within the modified interval is extracted and encapsulated according to preset fields to obtain a variational descriptor. The variational descriptor contains the starting offset, data length, and modified data content, recording metadata changes in a fine-grained incremental form. This eliminates the need to save the entire metadata page, reducing log storage and IO transmission load.

[0018] During the file system mounting phase, the host machine's physical alignment parameters are obtained in advance, and the guest log buffer is planned to ensure that subsequent log data is aligned with the underlying storage hardware. When metadata changes, lightweight variational descriptors are generated by differentially comparing dirty pages with the original mirror pages, recording only the local modification range of the page. This fully utilizes the buffer to achieve efficient log aggregation, reducing frequent IO interactions, and avoids write amplification caused by recording entire pages of logs, thus reducing the storage read and write overhead in the virtualization environment.

[0019] Step 120: Aggregate multiple temporally adjacent variational descriptors into log frames and compress them to obtain compressed log frames. Insert uncompressed index blocks between adjacent compressed log frames to form a log storage layout and store it in the log buffer. The uncompressed index blocks record the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frames.

[0020] By aggregating scattered and temporally adjacent variational descriptors into frames and compressing them uniformly, the number of single I / O interactions is reduced. Batch data streaming enhances compression gain, further reducing log storage capacity and persistent read / write overhead. For example, variational descriptors generated within the virtual machine are continuously collected, and multiple temporally adjacent variational descriptors are aggregated into the same frame buffer according to a preset capacity threshold, assembling into a complete log frame. Optionally, a single log frame capacity threshold of 16KB is set, and frame sealing is triggered when the total amount of variational descriptor data accumulated in the buffer reaches the threshold. After the log frame is assembled, the LZ4 compression algorithm is called to perform real-time compression processing on the variational data stream within the log frame, removing redundant information from the data stream, and finally outputting a compressed log frame. This avoids frequent system calls caused by processing variational descriptors independently, improves compression efficiency by utilizing batch data, effectively reduces the storage space occupied by log data in virtualization scenarios, and facilitates subsequent unified persistence, transmission, and playback recovery by log frame unit.

[0021] Embedding uncompressed index blocks between compressed log frames enables rapid location of target change logs based on inode numbers, avoiding complete decompression of the entire log stream and significantly improving log retrieval efficiency during virtual machine restart and recovery. Uncompressed index blocks are stored independently of compressed log frames and do not participate in the LZ4 compression process. The index block uses the inode number as the core search keyword, associating it with all variational descriptors within the corresponding interval to establish a mapping relationship between files and change log intervals, serving as a fast entry point for random retrieval. For example, in a continuously written compressed log stream, uncompressed index blocks as defined in this embodiment are inserted between two adjacent compressed log frames at preset fixed capacity intervals. The index block records the file inode numbers corresponding to the variational descriptors contained in the current interval's compressed log frames. The inode numbers are taken from the guest file system's native metadata and are distinct from the transaction sequence numbers. Optionally, an uncompressed index block is inserted every eight compressed log frames. The index block records the inode numbers corresponding to all changes within the interval, such as file identifiers like inode=1024 or inode=2048. This forms the physical arrangement of compressed data frames and uncompressed index points, which are then stored persistently in the log buffer. This physical arrangement is compatible with the clustered storage characteristics of virtual disks. When virtual machines restart on a large scale to perform data recovery, the log interval corresponding to the target file can be directly located based on the uncompressed index blocks, without having to decompress the entire log stream layer by layer, providing an efficient log retrieval entry point.

[0022] Step 130: Receive the persistence signal sent by the host machine, concatenate the compressed log frames in the log buffer into a batch write request that conforms to the physical alignment parameters, and send it to the host machine for storage.

[0023] The persistence signal is actively issued by the host virtualization transport layer as a synchronous trigger instruction for the guest machine to write logs to disk. It coordinates the timing of data flushing between the guest machine and the host machine, ensuring that virtual machine change logs are stably and persistently stored in a unified time sequence, avoiding out-of-order log writing and continuous expansion of the memory buffer. Driven by the persistence signal issued by the host machine, batch disk writes are performed, concatenating discrete compressed log frames into batch IO requests that meet physical alignment requirements. This reduces multiple small-scale IO interactions, alleviates write amplification issues caused by virtualization layer relocation, and improves cross-virtual machine data persistence throughput and storage efficiency.

[0024] For example, the client monitors the synchronization commands issued by the virtualization transport layer in real time, captures the persistence signals sent by the host, and marks the ready compressed log frames in the log buffer as pending commit. Then, multiple pending compressed log frames are concatenated in the commit queue to generate a batch write request matching the host's physical sector alignment parameters. This batch write request is sent to the host along with a FLUSH request. After the host returns a successful confirmation for the FLUSH request, confirming the data persistence is complete, the corresponding original image page in the client's memory is updated, and the space occupied by the data already written to disk in the log buffer is released. This client-host collaborative flushing mechanism avoids the frequent I / O overhead caused by independent writing of scattered log frames, reduces write amplification during virtualization forwarding, and ensures the timing consistency and storage access performance of persistent change log storage.

[0025] Using the persistent signal issued by the host machine as the trigger condition for log write-to-disk synchronization, compressed log frames in the scheduling buffer are uniformly arranged. Scattered log frames are spliced ​​and assembled into batch write requests that meet the physical alignment parameters before being sent to the host machine storage. On the one hand, this enables timing coordination between the client and the host machine for write-to-disk operations, preventing data accumulation in the buffer. On the other hand, by batch IO merging and physical boundary alignment, fragmented IO interactions are reduced, and the write amplification overhead caused by data relocation in the virtualization layer is reduced, effectively improving log persistence throughput performance. At the same time, the FLUSH acknowledgment mechanism ensures reliable log write-to-disk operation, maintaining the timing integrity and data consistency of virtual machine change log storage.

[0026] Step 140: When the virtual machine crashes and restarts, read the uncompressed index block, locate the uncommitted transactions before the crash, and overwrite the original metadata block with the incremental data of the corresponding partition descriptor according to the inode number recorded in the uncompressed index block to perform file system recovery mounting.

[0027] Virtual machine crashes and restarts are a typical abnormal scenario in virtualization operations and maintenance. When a virtual machine unexpectedly crashes, file system transactions that have not yet been persisted in memory are lost, and only partially completed change logs remain on the disk. Traditional recovery methods require sequentially and completely traversing the entire log stream for decompression and replay, which is time-consuming and cannot meet the needs of large-scale virtualization clusters to quickly resume business operations.

[0028] By leveraging uncompressed index blocks to quickly locate uncommitted transactions and accurately matching change records based on inode numbers, this approach abandons the serial scanning and replay mode of the entire log and utilizes multi-core parallel recovery, effectively shortening the file system mount readiness time after a crash and adapting to rapid recovery scenarios for large-scale virtual machine clusters. For example, after a virtual machine crash restart process is triggered, the log recovery engine within the mount process directly reads the uncompressed index block at the end of the log and uses a non-linear scanning method to quickly locate uncommitted transactions left over from before the crash execution interruption. Multi-core hardware resources allocated to the virtual machine are scheduled, and based on the file inode number recorded in the uncompressed index block, modulo sharding is performed on the variational tuples in the variational descriptors associated with the uncommitted transactions. The sharding tasks are distributed and mapped to different virtual processor cores. Each virtual processor core relies on a completion counter to achieve execution synchronization, and in parallel, the incremental data carried by the variational descriptors is accurately overwritten into the corresponding original metadata blocks of the file system. After all changes are applied, file system consistency verification is completed and recovery mount is performed.

[0029] The recovery process relies on uncompressed index blocks to achieve random navigation of logs, eliminating the need for complete decompression and sequential traversal of the entire log stream, significantly reducing transaction retrieval overhead. Inode sharding enables multi-core parallel scheduling of recovery tasks, addressing the slow pain point of traditional serial log replay recovery. It can quickly repair crashed and interrupted file system transactions, ensure metadata consistency, support virtual machine file systems to be mounted and ready in seconds, and significantly improve business recovery efficiency in virtualization cluster failure scenarios.

[0030] This embodiment obtains the physical alignment parameters from the host machine during the virtual machine file system mounting phase, divides the log buffer in the guest memory, and compares the modified dirty page with the original image page in the guest memory when the virtual machine kernel triggers metadata page modification to determine the starting offset and data length of the metadata modification. It then extracts the data within the modification interval to generate variational descriptors. Multiple temporally adjacent variational descriptors are aggregated into log frames and compressed to obtain compressed log frames. Uncompressed index blocks are inserted between adjacent compressed log frames to form a log storage layout stored in the log buffer. The uncompressed index blocks record the inode numbers of the files corresponding to the variational descriptors in the adjacent compressed log frames. Upon receiving a persistence signal from the host machine, the compressed log frames in the log buffer are concatenated into a batch write request conforming to the physical alignment parameters and sent to the host machine for storage. When the virtual machine crashes and restarts, the uncompressed index blocks are read to locate uncommitted transactions before the crash. Based on the inode numbers recorded in the uncompressed index blocks, the incremental data corresponding to the variational descriptors is overwritten onto the original metadata blocks to perform file system recovery mounting. By constructing a compressed log storage structure carrying inode indexes, and combining it with host machine collaborative alignment disk writing and multi-core parallel overwriting mechanisms, low-redundancy log recording of file system metadata and rapid crash recovery are achieved in a virtualized environment.

[0031] Example 2 Figure 2 This is a flowchart of a log storage and fast recovery method for a virtual machine file system provided in Embodiment 2 of the present invention. This embodiment is an optimization based on the above embodiment. The steps include: dividing a log buffer in the guest memory; when the virtual machine kernel triggers metadata page modification, comparing the modified dirty page with the original mirror page in the guest memory to determine the starting offset and data length of the metadata modification; extracting the modification interval data to generate a variational descriptor; calculating a checksum for the variational descriptor using the CRC32 algorithm; and writing the global monotonically increasing sequence number into the variational descriptor.

[0032] See Figure 2 Log storage and fast recovery methods for virtual machine file systems include: Step 210: During the virtual machine file system mounting phase, obtain the physical alignment parameters on the host side, divide the log buffer in the guest memory, and start the global monotonically increasing sequence number. When the virtual machine kernel triggers metadata page modification, compare the dirty page after the metadata page modification with the original image page in the guest memory to determine the starting offset and data length of the metadata modification, extract the data of the modification interval to generate a variational descriptor, calculate the checksum of the variational descriptor using the CRC32 algorithm, and write the global monotonically increasing sequence number into the variational descriptor.

[0033] In a virtualized environment, multiple concurrent metadata change transactions can occur. Without a unified time sequence identifier, it's difficult to accurately distinguish the order of transactions during log replay and crash recovery, potentially leading to out-of-order metadata overwriting. Furthermore, variational descriptors may experience bit errors and data truncation during cross-layer transmission and persistent storage between the client and host machines, causing data corruption during recovery. Therefore, it's necessary to introduce a globally monotonically increasing sequence number to construct a transaction time sequence benchmark, rely on CRC32 checksums for data integrity verification, and embed the sequence number into the variational descriptor. This integrates time sequence marking and integrity verification for each change record, ensuring orderly log collection, reliable transmission, and accurate replay.

[0034] During the mounting phase, a globally monotonically increasing sequence number is synchronously started, establishing a logical timing benchmark for subsequent transactions based on the virtual machine clock frequency. After the virtual machine completes the file system mounting initialization, the kernel log module initializes the sequence number counter, initially setting the sequence number to 1. Whenever a new variational descriptor is generated, the counter performs an atomic increment operation. Relying on the virtual machine's high-precision clock tick, the sequence number is guaranteed to be strictly monotonically increasing, preventing duplicate sums and rollback. When the virtual machine kernel triggers a metadata page modification, after generating the original payload of the variational descriptor through differential comparison between the dirty page and the original image page, a verification binding process is executed. This process extracts all payload content, including the internal offset, length, and incremental binary data of the variational descriptor, and calculates a 32-bit checksum using the standard CRC32 polynomial algorithm. Subsequently, the latest globally monotonically increasing sequence number is read, and the sequence number and CRC32 checksum are written together into the preset header field of the variational descriptor, completing the association binding of the sequence number, checksum, and changed payload. For example, the variational descriptor corresponding to the first metadata change is written with sequence number Seq=1 and CRC32 checksum 0x582c1fa3. The next change is automatically assigned Seq=2 and a new CRC32 value is calculated independently. Throughout the process of the variational descriptor being transmitted to the host machine via the virtualization front-end driver and stored in the log buffer, the receiving end can quickly identify data with transmission distortion and storage corruption based on the CRC32 checksum. At the same time, the built-in monotonic sequence number distinguishes the order of transactions, preventing out-of-order execution of transactions during log frame aggregation and parallel recovery from crashes, and ensuring that metadata changes are replayed in the actual order of occurrence.

[0035] Step 220: Aggregate multiple temporally adjacent variational descriptors into log frames and compress them to obtain compressed log frames. Insert uncompressed index blocks between adjacent compressed log frames to form a log storage layout and store it in the log buffer. The uncompressed index blocks record the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frames.

[0036] Step 230: Receive the persistence signal sent from the host machine, concatenate the compressed log frames in the log buffer into a batch write request that conforms to the physical alignment parameters, and send it to the host machine for storage.

[0037] Step 240: When the virtual machine crashes and restarts, read the uncompressed index block, locate the uncommitted transactions before the crash, and overwrite the original metadata block with the incremental data of the corresponding partition descriptor according to the inode number recorded in the uncompressed index block to perform file system recovery mounting.

[0038] This embodiment obtains the physical alignment parameters from the host machine during the virtual machine file system mounting phase, divides the log buffer in the guest memory, and starts a globally monotonically increasing sequence number. When the virtual machine kernel triggers metadata page modification, the modified dirty page is compared with the original image page in the guest memory to determine the starting offset and data length of the metadata modification. Data within the modification interval is extracted to generate variational descriptors. The checksum of the variational descriptors is calculated using the CRC32 algorithm, and the globally monotonically increasing sequence number is written into the variational descriptors. Multiple temporally adjacent variational descriptors are aggregated into log frames and compressed to obtain a compressed log file. Log frames are created by inserting uncompressed index blocks between adjacent compressed log frames to form a log storage layout stored in the log buffer. The uncompressed index blocks record the inode numbers of the files corresponding to the variational descriptors in the adjacent compressed log frames. Upon receiving a persistence signal from the host machine, the compressed log frames in the log buffer are concatenated into batch write requests conforming to physical alignment parameters and sent to the host machine for storage. When a virtual machine crashes and restarts, the uncompressed index blocks are read to locate uncommitted transactions before the crash. Based on the inode numbers recorded in the uncompressed index blocks, incremental data corresponding to the variational descriptors is overwritten onto the original metadata blocks, and the file system is restored and mounted. Using this method, the logical timing of metadata transactions is established based on the globally monotonically increasing sequence number embedded with the variational descriptors. Combined with CRC32 checksums, the data integrity of variational descriptor cross-layer transmission and persistence is ensured, achieving lightweight, orderly collection, efficient storage, and reliable, orderly crash recovery of metadata change logs.

[0039] Example 3 Figure 3 This is a schematic diagram of the log storage and fast recovery device for virtual machine file systems provided in Embodiment 3 of the present invention, as shown below. Figure 3As shown, the device includes: The comparison and extraction module 310 is used to obtain the physical alignment parameters on the host side during the virtual machine file system mounting stage, divide the log buffer in the guest memory, and compare the dirty page after the metadata page modification with the original image page in the guest memory when the virtual machine kernel triggers the metadata page modification, determine the starting offset and data length of the metadata modification, and extract the data of the modification interval to generate a variational descriptor. The compression insertion module 320 is used to aggregate multiple temporally adjacent variational descriptors into a log frame and perform compression processing to obtain a compressed log frame. An uncompressed index block is inserted between adjacent compressed log frames to form a log storage layout and store it in the log buffer. The uncompressed index block records the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frame. The storage module 330 is used to receive persistent signals sent from the host machine, concatenate the compressed log frames in the log buffer into batch write requests that conform to the physical alignment parameters, and send them to the host machine for storage. The location recovery module 340 is used to read the uncompressed index block when the virtual machine crashes and restarts, locate the uncommitted transactions before the crash, and overwrite the original metadata block with the incremental data of the corresponding partition descriptor according to the inode number recorded in the uncompressed index block to perform file system recovery mounting.

[0040] The log storage and fast recovery device for virtual machine file systems provided in this embodiment obtains the physical alignment parameters from the host machine during the virtual machine file system mounting phase, divides the log buffer in the guest memory, and compares the dirty page after the metadata page modification with the original mirror page in the guest memory when the virtual machine kernel triggers metadata page modification to determine the starting offset and data length of the metadata modification. It then extracts the data of the modification interval to generate variational descriptors. Multiple temporally adjacent variational descriptors are aggregated into log frames and compressed to obtain compressed log frames. Uncompressed index blocks are inserted between adjacent compressed log frames to form a log storage layout stored in the log buffer. The uncompressed index blocks record the inode numbers of the files corresponding to the variational descriptors in the adjacent compressed log frames. The device receives a persistence signal from the host machine, concatenates the compressed log frames in the log buffer into a batch write request that conforms to the physical alignment parameters, and sends it to the host machine for storage. When the virtual machine crashes and restarts, the uncompressed index blocks are read to locate uncommitted transactions before the crash. Based on the inode numbers recorded in the uncompressed index blocks, the incremental data corresponding to the variational descriptors is overwritten onto the original metadata blocks to perform file system recovery mounting. By constructing a compressed log storage structure carrying inode indexes, and combining it with host machine collaborative alignment disk writing and multi-core parallel overwriting mechanisms, low-redundancy log recording of file system metadata and rapid crash recovery are achieved in a virtualized environment.

[0041] Based on the above embodiments, the comparison and extraction module includes: The physical alignment parameter acquisition unit is used to identify whether the underlying storage is a virtualization backend by using hypercall instructions or querying the virtual bus interface during the virtual machine file system mounting stage. When the underlying storage is a virtualization backend, the physical alignment parameters of the host machine are acquired. The application area unit is used to apply for a contiguous area in the guest memory as a log buffer, and the memory mapping between the log buffer and the host machine is established through the virtualization input / output ring mechanism; The variational descriptor extraction unit is used to locate the dirty page after the metadata page modification based on the dirty page bitmap when the virtual machine kernel triggers the metadata page modification, compare the dirty page after the metadata page modification with the original image page in the guest memory, determine the starting offset and data length of the metadata modification, extract the data of the modification interval to generate a variational descriptor, and the variational descriptor includes the starting offset, data length and modified data content.

[0042] Based on the above embodiments, the compression insertion module includes: The LZ4 compression unit is used to aggregate multiple temporally adjacent variational descriptors into a log frame according to a capacity threshold, and call the LZ4 compression algorithm to compress the variational data stream in the log frame to obtain a compressed log frame.

[0043] Based on the above embodiments, the storage module includes: The persistent signal receiving unit is used to monitor the synchronization instructions issued by the virtualization transport layer in real time, receive persistent signals sent by the host machine, and mark the ready compressed log frames in the log buffer as pending submission. The request sending unit is used to concatenate the beginning and end of multiple compressed log frames marked as pending submission, generate a batch write request that matches the physical alignment parameter requirements, send the batch write request to the host machine, and send a FLUSH request to the host machine. The image page update and space release unit is used to update the original image page in the client memory and release the corresponding space in the log buffer after receiving a successful confirmation from the host machine in response to the FLUSH request.

[0044] Based on the above embodiments, the location recovery module includes: The uncommitted transaction locator is used to read the uncompressed index block at the end of the log when the virtual machine crashes and restarts, and to locate the uncommitted transactions generated before the crash using a non-linear scanning method. The modulo sharding unit is used to call the multi-core hardware resources allocated by the virtual machine, and to perform modulo sharding on the variational tuples in the variational descriptors associated with the uncommitted transaction based on the file inode number recorded in the uncompressed index block, and map them to different virtual processor cores. The overwrite and restore mount unit is used by each virtual processor core to synchronously and in parallel overwrite the incremental data corresponding to the variational descriptor onto the original metadata block of the file system through the completion counter, so as to restore the file system mount.

[0045] The log storage and fast recovery device for virtual machine file systems provided in this embodiment of the invention can execute the log storage and fast recovery method for virtual machine file systems provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0046] Example 4 Figure 4 This is a schematic diagram of the structure of a terminal provided in Embodiment 4 of the present invention. Figure 4 A block diagram is shown that is suitable for implementing an exemplary terminal 12 of the present invention. Figure 4 The terminal 12 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0047] like Figure 4 As shown, terminal 12 is presented in the form of a general-purpose computing terminal. The components of terminal 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0048] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0049] Terminal 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by terminal 12, including volatile and non-volatile media, removable and non-removable media.

[0050] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. Terminal 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0051] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0052] Terminal 12 can also communicate with one or more external terminals 14 (e.g., keyboard, pointing terminal, display 24, etc.), one or more terminals that enable a user to interact with terminal 12, and / or any terminal (e.g., network card, modem, etc.) that enables terminal 12 to communicate with one or more other computing terminals. This communication can be performed via input / output (I / O) interface 22. Furthermore, terminal 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of terminal 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal 12, including but not limited to: microcode, terminal drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0053] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the log storage and fast recovery method for virtual machine file systems provided in the embodiments of the present invention.

[0054] Example 5 Embodiment 5 of the present invention also provides a medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the log storage and fast recovery method for virtual machine file systems as described in any of the above embodiments.

[0055] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0056] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0057] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0058] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or device. In cases involving a remote computer, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0059] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for log storage and fast recovery of virtual machine file systems, characterized in that, include: During the virtual machine file system mounting phase, the physical alignment parameters on the host side are obtained, a log buffer is partitioned in the guest memory, and when the virtual machine kernel triggers metadata page modification, the dirty page after metadata page modification is compared with the original image page in the guest memory to determine the starting offset and data length of metadata modification, and the data of the modification interval is extracted to generate a variational descriptor. Multiple temporally adjacent variational descriptors are aggregated into log frames and compressed to obtain compressed log frames. Uncompressed index blocks are inserted between adjacent compressed log frames to form a log storage layout stored in the log buffer. The uncompressed index blocks record the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frames. Receive the persistence signal sent by the host machine, concatenate the compressed log frames in the log buffer into a batch write request that conforms to the physical alignment parameters, and send it to the host machine for storage; When the virtual machine crashes and restarts, the uncompressed index block is read to locate the uncommitted transactions before the crash. Based on the inode number recorded in the uncompressed index block, the incremental data of the corresponding partition descriptor is overwritten to the original metadata block to restore and mount the file system.

2. The method according to claim 1, characterized in that, During the virtual machine file system mounting phase, the process of obtaining the physical alignment parameters from the host machine and partitioning the log buffer in the guest memory includes: During the virtual machine file system mounting phase, the underlying storage is identified as a virtualization backend by using hypercall instructions or querying the virtual bus interface. If the underlying storage is a virtualization backend, the physical alignment parameters of the host machine are obtained. A contiguous region is allocated in the guest memory as a log buffer, and the log buffer is mapped to the host machine's memory through a virtualization input / output ring mechanism.

3. The method according to claim 1, characterized in that, When the virtual machine kernel triggers a metadata page modification, the modified dirty page is compared with the original image page in the guest memory to determine the starting offset and data length of the metadata modification, and the modified interval data is extracted to generate a variational descriptor, including: When the virtual machine kernel triggers a metadata page modification, the dirty page after the metadata page modification is located based on the dirty page bitmap. The dirty page after the metadata page modification is compared with the original image page in the guest memory to determine the starting offset and data length of the metadata modification. The data of the modification interval is extracted to generate a variational descriptor. The variational descriptor includes the starting offset, data length and modified data content.

4. The method according to claim 1, characterized in that, The process of aggregating multiple temporally adjacent variational descriptors into a log frame and compressing it to obtain a compressed log frame includes: Multiple temporally adjacent variational descriptors are aggregated into log frames according to a capacity threshold. The LZ4 compression algorithm is then used to compress the variational data stream in the log frames to obtain compressed log frames.

5. The method according to claim 1, characterized in that, The process of receiving the persistent signal from the host machine, concatenating the compressed log frames in the log buffer into a batch write request that conforms to the physical alignment parameters, and sending it to the host machine for storage includes: Real-time monitoring of synchronization commands issued by the virtualization transport layer, receiving persistent signals sent by the host machine, and marking the ready compressed log frames in the log buffer as pending submission. Multiple compressed log frames marked as pending submission are concatenated end to end to generate a batch write request that matches the physical alignment parameter requirements. The batch write request is then sent to the host machine, and a FLUSH request is sent to the host machine. After receiving a successful confirmation from the host machine regarding the FLUSH request, the original image page in the client's memory is updated, and the corresponding space in the log buffer is released.

6. The method according to claim 1, characterized in that, When a virtual machine crashes and restarts, the process involves reading the uncompressed index block, locating uncommitted transactions prior to the crash, and overwriting the original metadata block with incremental data corresponding to the corresponding partition descriptors based on the inode numbers recorded in the uncompressed index block, thus performing file system recovery mounting. This includes: When the virtual machine crashes and restarts, the uncompressed index block at the end of the log is read, and a non-linear scanning method is used to locate the uncommitted transactions generated before the crash. The virtual machine allocates multi-core hardware resources, and based on the file inode number recorded in the uncompressed index block, modulo-slices the variational tuples in the variational descriptors associated with the uncommitted transaction, and maps them to different virtual processor cores. Each virtual processor core performs file system recovery mounting by synchronously and in parallel overwriting the incremental data corresponding to the variational descriptor onto the original metadata block of the file system using a counter.

7. The method according to claim 1, characterized in that, The process of dividing a log buffer in the guest memory, comparing the modified dirty page with the original mirror page in the guest memory when the virtual machine kernel triggers metadata page modification, determining the starting offset and data length of the metadata modification, and extracting the modified interval data to generate a variational descriptor, further includes: Allocate a log buffer in the client's memory and start a globally monotonically increasing sequence number; When the virtual machine kernel triggers a metadata page modification, the dirty page after the metadata page modification is compared with the original image page in the guest memory to determine the starting offset and data length of the metadata modification, and extract the data of the modification interval to generate a variational descriptor. The checksum is calculated using the CRC32 algorithm for the variational descriptor, and the globally monotonically increasing sequence number is written into the variational descriptor.

8. A log storage and fast recovery device for virtual machine file systems, characterized in that, include: The comparison and extraction module is used to obtain the physical alignment parameters on the host side during the virtual machine file system mounting stage, divide the log buffer in the guest memory, and compare the dirty page after the metadata page modification with the original image page in the guest memory when the virtual machine kernel triggers the metadata page modification, determine the starting offset and data length of the metadata modification, and extract the data of the modification interval to generate a variational descriptor. The compression insertion module is used to aggregate multiple temporally adjacent variational descriptors into a log frame and perform compression processing to obtain a compressed log frame. An uncompressed index block is inserted between adjacent compressed log frames to form a log storage layout and store it in the log buffer. The uncompressed index block records the inode number of the file corresponding to the variational descriptor in the adjacent compressed log frame. The storage module is used to receive persistent signals from the host machine, concatenate the compressed log frames in the log buffer into batch write requests that conform to the physical alignment parameters, and send them to the host machine for storage. The location recovery module is used to read the uncompressed index block when the virtual machine crashes and restarts, locate the uncommitted transactions before the crash, and overwrite the original metadata block with the incremental data of the corresponding partition descriptor according to the inode number recorded in the uncompressed index block to perform file system recovery mounting.

9. A terminal, characterized in that, The terminal includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log storage and fast recovery method for virtual machine file systems as described in any one of claims 1-7.

10. A medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the log storage and fast recovery method for a virtual machine file system as described in any one of claims 1-7.

Images