Universal large-size data storage method and system

Abstract

The invention provides a general large-size data storage method and system, and relates to the technical field of computer storage. By adopting the Extent with the fixed size to manage the storage space, executing the asynchronous copy-on-write operation and implementing a double-buffer log mechanism, the problem of space waste caused by immediately copying a complete data block in large-size data storage is effectively solved, meanwhile, the persistence reliability of metadata change under the abnormal condition is guaranteed, and the storage efficiency is improved. The method has the advantages that the storage space utilization rate is increased, the data consistency guarantee capability is enhanced, and the write-in delay is reduced.

Description

Technical Field

[0001] This application relates to the field of computer storage technology, and in particular to a general large-size data storage method and system. Background Technology

[0002] In the field of computer storage technology, copy-on-write is a key data management strategy. Its basic principle is to trigger a copy operation when data is modified to optimize storage resource utilization and improve system efficiency. This technology has been widely applied in scenarios such as file system snapshot generation, virtualization image maintenance, and container environment management. However, with the continuous expansion of data scale, especially when facing the demand for large-scale data storage, existing copy-on-write storage systems have exposed several shortcomings. Traditional implementations must immediately copy the entire data block during the data writing process. This mechanism leads to a significant reduction in storage space utilization, and the problem of wasted storage resources is particularly prominent in high-frequency write or massive data scenarios. At the same time, the ability to guarantee data consistency is seriously insufficient. When encountering abnormal situations such as system power outages or unexpected crashes, existing mechanisms cannot ensure the complete persistence of metadata changes, which can easily lead to data corruption or loss. Regarding log management, current systems generally adopt a simple append-only log recording method, lacking effective space reclamation and compression mechanisms. This causes log files to accumulate and expand indefinitely, not only consuming a large amount of storage resources but also increasing the complexity of system recovery. Furthermore, in multi-user concurrent access environments, lock contention causes system performance to drop sharply with increasing concurrency. Existing storage solutions are often limited to specific platforms and cannot be flexibly deployed in diverse environments such as Linux and UEFI, further restricting the applicability of the technology. These technical bottlenecks collectively hinder the reliable application of copy-on-write mechanisms in modern large-scale storage systems. Summary of the Invention

[0003] This application provides a general large-size data storage method and system, which has the advantages of improving storage space utilization, enhancing data consistency assurance capabilities, and reducing write latency.

[0004] Firstly, the general large-size data storage method provided in this application adopts the following technical solution: A general large-size data storage method includes: In the core layer of the storage system, fixed-size Extents are used as the basic storage units to manage the storage space. Each Extent has a Map state, a Free state, or a Parent state. When a data write request is received, a copy-on-write (COW) operation is performed, including: checking the status of the target extent; if the target extent is in the parent state, allocating a new extent for the data and creating an asynchronous COW copy task to write the data to the newly allocated extent. For metadata change operations in the storage system, consistency is guaranteed through a double-buffered log mechanism. The double-buffered log mechanism includes: first, appending metadata changes to the memory log queue in the form of logs, and then periodically flushing the logs in memory to the disk in batches for persistence.

[0005] Optionally, the method of managing storage space using fixed-size Extents as basic storage units includes: The allocation status of all extents is managed using a red-black tree; Calculate the corresponding Extent ID based on the file offset of the data write request; A proximity-first allocation strategy is used to allocate new extent space for the data.

[0006] Optionally, the state of the target extent to be checked includes: If the target Extent is in a Free state, a new Extent is allocated directly, and the data is written to the newly allocated Extent, while the BIND log is recorded. If the target extent is in a Map state, the data is written directly to that extent.

[0007] Optionally, the creation of the asynchronous COW replication task includes: The COW replication task is placed in the background task queue. The COW replication task includes: copying data from the parent Extent to the newly allocated Extent, then writing the new data to the newly allocated Extent, and updating the Extent state to Map. The write request is returned immediately after the COW replication task is created, without waiting for the data replication to complete.

[0008] Optionally, the double-buffered logging mechanism also includes log replay and cleanup: During system initialization, the memory state of the storage system is reconstructed by replaying the logs on the disk; Perform log cleanup regularly to compress the current system state into a minimal log sequence, thereby freeing up storage space.

[0009] Optionally, the method employs a multi-level locking mechanism for concurrency control, the multi-level locking mechanism including: File tree read-write locks are used to protect file metadata and extent mappings, allowing multiple read operations to access concurrently; Task queue locks are used to manage background task scheduling; Atomic counters are used for write operation counting and state synchronization.

[0010] Optionally, the method achieves cross-platform deployment through a platform adaptation layer, which includes at least one of the following: Linux platform adaptation layer, used to implement block device drivers and BIO forwarding; The UEFI platform adaptation layer is used to implement EFI_BLOCK_IO_PROTOCOL adaptation. The user-space adaptation layer is used to provide pthread / glibc environment support.

[0011] Secondly, this application provides a general-purpose large-size data storage system, comprising: The management module is used to manage storage space in the core layer of the storage system, using fixed-size Extents as the basic storage units. Each Extent has a Map state, a Free state, or a Parent state. The Cow operation module is used to perform copy-on-write (COW) operations when a data write request is received. This includes: checking the status of the target Extent; if the target Extent is in the Parent state, allocating a new Extent for the data and creating an asynchronous COW copy task to write the data to the newly allocated Extent. The change module is used to perform metadata change operations on the storage system. Consistency is ensured through a double-buffered log mechanism. The double-buffered log mechanism includes: first, appending the metadata changes to the memory log queue in the form of logs, and then periodically flushing the logs in memory to the disk in batches for persistence.

[0012] Thirdly, this application provides a computer device, the device comprising: a memory and a processor, wherein the processor, when executing computer instructions stored in the memory, performs the method described above.

[0013] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0014] In summary, this application effectively solves the problem of wasted space caused by immediately copying complete data blocks in large-size data storage by using fixed-size Extents to manage storage space, performing asynchronous copy-on-write operations, and implementing a double-buffered log mechanism. At the same time, it ensures the persistence reliability of metadata changes under abnormal conditions, and has the advantages of improving storage space utilization, enhancing data consistency assurance capabilities, and reducing write latency. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application; Figure 2 This is a flowchart illustrating the first embodiment of the general large-size data storage method of this application; Figure 3 This is a structural block diagram of the first embodiment of the general-purpose large-size data storage system of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0017] Reference Figure 1 , Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application.

[0018] like Figure 1 As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. Optionally, the memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0019] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0020] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a general-purpose large-size data storage program.

[0021] exist Figure 1 In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in this application can be set in the computer device, and the computer device calls the general large-size data storage program stored in the memory 1005 through the processor 1001 and executes the general large-size data storage method provided in the embodiment of this application.

[0022] This application provides a general large-size data storage method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the general large-size data storage method of this application.

[0023] In this embodiment, the general large-size data storage method includes the following steps: Step S10: In the core layer of the storage system, fixed-size Extents are used as basic storage units to manage the storage space, where each Extent has a Map state, a Free state, or a Parent state.

[0024] It's important to note that low storage space utilization is a common problem in traditional copy-on-write storage systems. Specifically, data write operations trigger immediate replication of the entire data block, increasing storage overhead, especially in large-scale data scenarios where system resources are inefficiently utilized. Furthermore, in the event of power outages or other abnormal situations, the persistence mechanism for metadata changes is insufficient, making data consistency recovery difficult and prone to corruption or loss. Simultaneously, the log management mechanism lacks effective space control strategies, leading to continuously growing log files, increasing the complexity of system maintenance, and impacting the long-term stable operation of the storage system.

[0025] For example, in the specific scenario of virtual machine snapshot management on a cloud platform, when multiple users concurrently access a shared disk image, the system detects that the target extent is in the Parent state. Traditional methods immediately execute a copy operation for the entire extent, resulting in wasted storage space. In the event of a power failure, incomplete metadata changes cannot be correctly reconstructed through logs, leading to inconsistent snapshot data. Specifically, frequent metadata operations cause the log space to expand indefinitely, requiring additional system resources for log cleanup and reducing overall operational efficiency.

[0026] If the above problems are not addressed, the resource utilization of the storage system will continue to decrease, the risk to data integrity will increase, and the system's recovery capability after abnormal events will be limited, which may lead to service interruptions and data unavailability, thereby negatively impacting the reliability and maintainability of the storage system.

[0027] This embodiment provides a general method for large-size data storage. Fixed-size extents are used as the basic storage units. Each extent has a Map state, a Free state, or a Parent state to identify the allocation and usage of storage space. In practical applications, extent state management can be implemented using a bitmap structure, for example, by mapping the state of each extent using a single bit; or by using a linked list to organize extents in the same state together, facilitating state querying and allocation operations. This is primarily to achieve precise management of storage space, avoiding the real-time copying of full data blocks as in traditional copy-on-write. Furthermore, when a data write request is received, the state of the target extent is checked. If it is in the Parent state, a new extent is allocated, and an asynchronous copy-on-write (COW) task is created, and the data is written to the newly allocated extent. Specifically, the target extent's status can be checked by accessing the status register in memory; the allocation of a new extent can be selected based on a list of free blocks; the creation of asynchronous COW replication tasks can utilize the thread mechanism provided by the operating system to perform data replication in the background; this is mainly to achieve immediate response to write operations and alleviate performance bottlenecks in high-concurrency scenarios. Thus, write requests return immediately after task creation, without waiting for data replication to complete. Furthermore, metadata change operations are guaranteed for consistency through a double-buffered log mechanism. This mechanism includes appending metadata changes to a memory log queue in log form, and then periodically flushing the logs in memory to disk in batches. For example, the memory log queue can be implemented using a circular buffer structure; batch flushing of logs can be triggered based on time intervals or queue size thresholds; this is mainly to achieve efficient integration between memory buffering and disk persistence, ensuring data consistency recovery under abnormal conditions such as power outages. This embodiment, through the synergy of the above technical means, effectively solves the problems of low storage space utilization, difficulty in guaranteeing data consistency under abnormal conditions such as power outages, and complex log space management caused by copy-on-write operations in large-size data storage. Specifically, the stateful management of extents enables the system to accurately identify data sharing relationships based on real-time status; the asynchronous copy-on-write (COW) mechanism avoids the impact of write blocking on system throughput; and the double-buffered log mechanism achieves effective log space management by reducing disk I / O frequency and ensuring the reliability of the persistence process. As a preferred implementation, the combined design of the above features enables the storage system to achieve efficient and reliable data processing capabilities at the core layer.

[0028] In one implementation, the core layer of the storage system uses fixed-size extents as the basic storage units for space management. Each extent is assigned a Map, Free, or Parent state to indicate its current usage. When a data write request is received, the system performs a copy-on-write operation, specifically including checking the target extent's state. If the state is Parent, a new extent space is allocated for the target data, and an asynchronous COW replication task is created, with the data directly written to the newly allocated extent. Furthermore, metadata changes are guaranteed for consistency through a double-buffered log mechanism. Metadata changes are first appended to an in-memory log queue in log form, and then periodically batch-flushd to disk for persistence under predetermined conditions. The extent state management mechanism enables the system to accurately identify data sharing relationships; for example, the Parent state explicitly indicates that the extent is shared by multiple snapshots or versions, thus triggering a replication operation only on the shared portion during writes, avoiding the overhead of traditional full replication and improving storage space utilization efficiency. Specifically, when the target Extent is in the Parent state, the system immediately allocates a new Extent and starts an asynchronous replication task. Write requests can return immediately after task creation, without waiting for replication to complete. This effectively alleviates lock contention issues in high-concurrency scenarios. Simultaneously, the asynchronous task performs data replication in the background, eliminating the impact of write blocking on system throughput. Furthermore, the double-buffered log mechanism uses the aggregation characteristics of memory queues to merge multiple metadata changes into batch operations, reducing disk I / O frequency. During system initialization, the memory state is reconstructed by replaying the disk logs, ensuring data consistency recovery in abnormal situations such as power outages, while also preventing the log space from expanding indefinitely.

[0029] As a specific implementation, consider a scenario where the extent size is configured to 4MB. When processing a file write request, the system calculates the target extent ID based on the file offset, detects its status as Parent (indicating that the extent is shared by multiple snapshots), allocates a new extent space, and writes the data to that new space. Simultaneously, the created COW replication task is placed in a background task queue to perform the replication operation of the parent extent data. For metadata changes, such as an extent status update from Parent to Map, the relevant logs are appended to an in-memory log queue. When the queue reaches a preset capacity threshold or a timer is triggered, the logs are written to disk in batches. Thus, the system simplifies the log management process while ensuring data consistency.

[0030] The above technical solutions solve the problem of low storage space utilization caused by copy-on-write operations in large-size data storage, because the copy operation is triggered only when the target extent is in the parent state, avoiding unnecessary full data copying; the problem of data consistency not being guaranteed under abnormal conditions such as power outages is effectively addressed, thanks to the double-buffered log mechanism that supports log replay and state reconstruction through the connection between memory queues and disk persistence; the complexity of log space management is significantly reduced, and efficient compression and space release of logs are achieved through a periodic batch flushing strategy.

[0031] Specifically, in some of the above embodiments of this example, a fixed-size Extent is proposed as the basic storage unit for managing storage space. However, in its implementation, the management of Extent allocation status lacks efficient data structure support, resulting in increased time consumption for status query and update operations. At the same time, the calculation of Extent ID relies on inefficient traversal or mapping mechanisms, introducing additional metadata processing overhead. In addition, the space allocation strategy does not consider the principle of locality, which easily leads to storage fragmentation problems, thereby reducing the writing efficiency of large-size data and the overall space utilization, exacerbating the defects of low storage efficiency and concurrency performance bottleneck in the prior art.

[0032] To address this, this embodiment further proposes using fixed-size Extents as basic storage units to manage storage space, including: managing the allocation status of all Extents through a red-black tree; calculating the corresponding Extent ID based on the file offset of the data write request; and allocating new Extent space for data using a proximity-first allocation strategy.

[0033] Among them, red-black trees refer to a self-balancing binary search tree data structure, which can be implemented using a balanced tree algorithm implemented in the standard library or a custom node coloring mechanism. Its purpose is to ensure that the query, insertion, and deletion operations of Extent allocation status are completed in logarithmic time complexity by maintaining the height balance of the tree, thereby avoiding the performance bottleneck caused by linear search. The calculation of Extent ID refers to deriving a unique identifier based on the mathematical mapping relationship between file offset and fixed Extent size. It can be implemented using integer division or bitwise operations. Its purpose is to quickly locate the target Extent directly through the offset, eliminating redundant search steps and accelerating metadata processing. The proximity-first allocation strategy refers to prioritizing the physical region adjacent to the current logical location of the data when allocating new space. It can be implemented using address continuity detection or spatial proximity scoring mechanisms. Its purpose is to maintain data locality to reduce fragmentation accumulation and improve storage space continuity.

[0034] Specifically, the solution in this embodiment uses a red-black tree to efficiently index the extent allocation status, enabling state change operations to quickly locate the target node. Simultaneously, the extent ID calculation mechanism based on file offsets directly establishes a mapping between logical addresses and physical storage units, avoiding the overhead of traditional traversal lookups. Furthermore, a proximity-first allocation strategy dynamically evaluates physical proximity during space allocation, prioritizing free extents adjacent to the current operation area. These three elements work together to form a complete closed loop for storage space management: the red-black tree provides underlying data structure support, the ID calculation mechanism optimizes metadata processing efficiency for write paths, and the allocation strategy reduces fragmentation risks from a spatial layout perspective, collectively ensuring low-latency, high-throughput storage operations in large-scale data scenarios.

[0035] As a specific implementation method, the solution in this embodiment is implemented as follows: During the storage system initialization phase, a red-black tree structure is configured to manage the status information of all Extents, where each node stores the starting address, status identifier, and associated pointer of the Extent; when a data write request is received, the system directly calculates the corresponding Extent ID based on the quotient of the file offset divided by a fixed Extent size (e.g., 4MB), without traversing the metadata table; when allocating new Extent space, the system prioritizes scanning contiguous free block regions with physical addresses. If adjacent free blocks exist, they are allocated immediately; otherwise, the system falls back to the global free list for selection. In practical applications, the red-black tree can be implemented based on the rbtree provided by the Linux kernel. The ExtentID calculation uses bitwise operations instead of division to improve efficiency, while the proximity-first strategy automatically triggers local allocation logic when a contiguous free region is detected.

[0036] Through the above solution, this embodiment effectively reduces the latency of Extent status query and update, reduces metadata processing overhead, and significantly alleviates the storage fragmentation problem by maintaining data locality, thereby improving the writing efficiency of large-size data and the overall space utilization, providing a reliable technical foundation for solving the low storage efficiency and concurrency performance bottleneck in the existing technology.

[0037] In practical applications, some of the above-described embodiments of this example propose checking the state of the target extent to perform data write operations. However, in its implementation, the processing logic for the Free state and the Map state is missing, which may lead to wasted storage space or data consistency risks. Specifically, when the target extent is in the Free state, the lack of an explicit allocation mechanism can easily lead to invalid writes, while when it is in the Map state, the lack of a direct write strategy may cause redundant operations or state conflicts.

[0038] In response, this embodiment further proposes steps for checking the state of the target extent, including: If the target Extent is in a Free state, a new Extent is allocated directly, and the data is written to the newly allocated Extent, while the BIND log is recorded. If the target extent is in a Map state, the data is written directly to that extent.

[0039] The Free state refers to the idle state of an Extent, indicating that the Extent is not currently occupied by any data. This can be implemented using status flags or bitmaps, and its purpose is to quickly identify allocatable space and avoid writing operations to invalid areas. The Map state can be understood as the mapped state of an Extent, indicating that the Extent has been allocated and contains valid data. This can be implemented using status registers or metadata fields, and its purpose is to indicate directly accessible data areas, ensuring the accuracy of write operations. The BIND log refers to the log entries that record metadata changes. It can be implemented using an append-only log structure, and its purpose is to ensure the atomicity and recoverability of metadata changes, providing a guarantee for data consistency in abnormal situations.

[0040] Specifically, the solution in this embodiment uses a state check mechanism to dynamically select the write path based on the actual state of the target Extent: when a Free state is detected, the system immediately allocates a new Extent and performs a write operation, while simultaneously recording the BIND log to capture metadata change events; when a Map state is detected, the system directly writes the data to the target Extent, eliminating the need for additional allocation or copying steps. This state-based branching logic ensures the minimized path of the write operation, effectively reducing unnecessary resource consumption, and maintains metadata consistency through the synchronous recording of the BIND log, thereby forming a closed-loop control between state identification and operation execution.

[0041] As a specific implementation method, the solution in this embodiment is implemented as follows: When the storage system processes a data write request, the metadata management module first queries the status information of the target Extent; if the status indication is Free, a new Extent space is allocated, the data is written to the new space, and a BIND log is generated to record the allocation event; if the status indication is Map, the new data is directly overwritten into the Extent. This status information can be maintained by a metadata table in memory, where the status field of each Extent is updated through atomic operations, such as using binary encoding to represent different states, to ensure the reliability of state transitions.

[0042] Through the above solution, this embodiment effectively avoids the risk of invalid writes in the Free state and the problem of redundant operations in the Map state, significantly improves storage space utilization and data write efficiency, and at the same time ensures the atomicity of metadata changes and the data consistency of the system under abnormal conditions.

[0043] Step S20: When a data write request is received, perform a copy-on-write (COW) operation, including: checking the status of the target extent; if the target extent is in the parent state, allocate a new extent for the data and create an asynchronous COW copy task to write the data to the newly allocated extent.

[0044] In specific implementations, in some of the above embodiments of this example, a copy-on-write (COW) operation is proposed to handle data modification. However, in its implementation, the COW copy task needs to be executed synchronously, which causes write requests to wait for the data copy to complete, thereby increasing write latency and reducing the system's concurrent processing capability. Especially in the scenario of writing large amounts of data, it significantly affects the system's response speed and resource utilization efficiency.

[0045] In response, this embodiment further proposes the creation of asynchronous COW replication tasks, including: placing the COW replication task into a background task queue; the COW replication task includes: copying data from the parent Extent to the newly allocated Extent, then writing the new data into the newly allocated Extent, and updating the Extent's state to Map; the write request returns immediately after the COW replication task is created, without waiting for the data replication to complete.

[0046] In this context, a Copy-on-Write (COW) task refers to an asynchronous data copying operation triggered during data modification. It can be implemented using a separate worker thread or a task scheduler, aiming to decouple the copying operation from the foreground write process and prevent user requests from being blocked for extended periods. The background task queue can be understood as an ordered data structure used to temporarily store and schedule COW copying tasks. It can be implemented using a first-in-first-out (FIFO) queue or a priority queue mechanism, aiming to rationally allocate system resources and ensure the orderly execution of multiple concurrent tasks. Copying data from the parent Extent refers to the process of reading the original data from the parent Extent and transferring it to the newly allocated Extent. This can be implemented using direct memory access or zero-copy technology, aiming to safely retain a copy of the data for subsequent writes. It provides a foundation for consistency; writing new data to a newly allocated Extent can be understood as writing the modified content to the new space after the copy operation is completed. This can be implemented using buffered writes or direct I / O, with the aim of ensuring the integrity and correctness of the new data; updating the Extent state to Map means marking the Extent's state as available. This can be implemented using atomic operations or state machine mechanisms, with the aim of reflecting the validity of the storage unit in a timely manner and avoiding state management chaos; the write request returns immediately after the COW copy task is created, meaning that the write operation is completed and responded to the moment the task is created. This can be implemented using a non-blocking I / O model, with the aim of improving system throughput and alleviating lock contention pressure in high-concurrency scenarios.

[0047] Specifically, this embodiment decouples write requests from the data copying process by asynchronously processing COW copying tasks. When the system detects that the target Extent is in the Parent state, it immediately creates a COW copying task and adds it to the background task queue. Simultaneously, write requests are allowed to return immediately after task creation, without waiting for copying to complete. The background task queue schedules COW copying tasks according to a predetermined strategy. During task execution, each task sequentially copies data from the parent Extent, writes new data, and updates the Extent's state. This design ensures that foreground write operations are no longer affected by copying delays, while background tasks efficiently process data copying using idle system resources, thereby optimizing CPU and I / O resource utilization. Furthermore, the sequential execution of tasks and the timeliness of state updates ensure the atomicity and consistency of data modifications.

[0048] As a preferred embodiment, the solution of this embodiment is implemented as follows: The background task queue is implemented as a priority-based queue structure, in which high-priority tasks are scheduled and executed first; the COW replication task is handled by threads in the system's worker thread pool. After the thread retrieves the task from the queue, it first reads the data block of the parent Extent and copies it to the newly allocated Extent, then writes the modified data to the Extent, and finally updates the Extent's state to Map through atomic operations; during this process, the write request returns a confirmation signal to the user immediately after the task is created, and the user does not need to wait for the background operation to complete, while the system efficiently performs data replication during idle periods.

[0049] Through the above solution, this embodiment effectively reduces write latency, improves the system's concurrent processing capability in large-size data writing scenarios, and enhances response speed and resource utilization efficiency.

[0050] Step S30: For metadata change operations of the storage system, consistency is guaranteed through a double-buffered log mechanism. The double-buffered log mechanism includes: first, appending the metadata changes to the memory log queue in the form of logs, and then periodically flushing the logs in memory to the disk in batches for persistence.

[0051] It should be noted that in some of the above implementations of this embodiment, a double-buffered log mechanism is proposed to ensure metadata consistency. However, in its implementation, the log space is prone to infinite expansion, and there is a lack of effective log organization and compression mechanisms, resulting in wasted storage space. At the same time, during system initialization, the memory state cannot be reconstructed through the log, affecting system recovery efficiency.

[0052] In response, this embodiment further proposes a double-buffered logging mechanism that also includes log replay and organization: During system initialization, the memory state of the storage system is reconstructed by replaying the logs on the disk; Perform log cleanup regularly to compress the current system state into a minimal log sequence, thereby freeing up storage space.

[0053] Log replay refers to the process of reconstructing the memory state of the storage system before a crash by parsing persistent log records on the disk. This can be achieved by sequentially scanning log files and replaying them according to timestamps. The purpose is to ensure that the system can accurately restore the memory state without relying on external data sources after an abnormal restart. Log defragmentation can be understood as the operation of identifying and merging redundant log entries. This can be achieved by using checkpoint-based compression algorithms or incremental merging strategies. The purpose is to reduce the physical storage space occupied by logs. Minimal log sequence refers to the simplest log set that can completely represent the current system state. This can be achieved by retaining key state change records and removing intermediate redundant operations. The purpose is to prevent resource waste caused by the infinite growth of logs.

[0054] Specifically, the solution in this embodiment automatically triggers the log replay process during the system initialization phase. It replays metadata change operations in chronological order using the complete and persistent log sequence on the disk, accurately reconstructing the file mapping relationship and Extent state information in memory, thereby achieving crash recovery without external intervention. At the same time, through periodically executed log tidying operations, it dynamically scans the historical log sequence, identifies old state records that have been overwritten by subsequent operations, and merges these redundant entries into a single latest state representation, forming a compact log sequence. This process runs in the background and does not affect normal business operations, ensuring efficient utilization of log space and long-term stable operation of the system.

[0055] As a specific implementation, the solution of this embodiment is implemented as follows: When the system starts, the initialization module first loads the log file on the disk, parses each metadata change record in chronological order and applies it to the memory red-black tree structure to complete the reconstruction of the mapping relationship from file offset to Extent ID; During system operation, the log consolidator is periodically woken up to scan the current log sequence, identify old log entries that have been covered by new BIND logs, and merge these redundant records into the minimum log sequence containing only the final state, thereby releasing the occupied storage space.

[0056] Through the above solution, this embodiment effectively solves the problem of storage resource waste caused by the unlimited expansion of log space and significantly improves the sustainability of the log mechanism. At the same time, the complete memory state can be quickly reconstructed during system initialization, which greatly shortens the anomaly recovery time and ensures data consistency and system reliability.

[0057] Specifically, in some of the above embodiments of this example, a general large-size data storage method is proposed. However, in its implementation process, there is a lack of an effective concurrency control mechanism, which leads to problems such as increased lock contention and significant system performance degradation in multi-user concurrent access scenarios. In particular, when processing a large number of write requests, core operations such as Extent status checks and metadata updates become performance bottlenecks due to global lock contention.

[0058] To address this, this embodiment further proposes using a multi-level locking mechanism for concurrency control. The multi-level locking mechanism includes: File tree read-write locks are used to protect file metadata and extent mappings, allowing multiple read operations to access concurrently; Task queue locks are used to manage background task scheduling; Atomic counters are used for write operation counting and state synchronization.

[0059] Among them, the file tree read-write lock is a synchronization mechanism that allows multiple read operations to be executed concurrently, but write operations require exclusive access. It can be implemented using read-write lock primitives provided by the operating system kernel or read-write lock structures implemented in user space. Its purpose is to protect file metadata and extent mapping relationships while maximizing the concurrency of read operations. The task queue lock can be understood as a mutex lock used to protect the background task queue data structure. It can be implemented using standard mutex lock mechanisms or spin locks. Its purpose is to ensure the atomicity and consistency of task scheduling operations. The atomic counter specifically refers to a counter implemented using atomic instructions provided by the processor. It can be implemented using hardware-supported atomic operation library functions or compiler-built-in atomic types. Its purpose is to efficiently perform write operation counting and state synchronization without the intervention of locks.

[0060] Specifically, this embodiment uses a file tree read-write lock to allow multiple read operations to simultaneously access file metadata and extent mappings, thus avoiding mutual exclusion waiting between read operations in read-intensive scenarios. A task queue lock independently manages the scheduling of background COW replication tasks, preventing lock resource conflicts between foreground write operations and background tasks. An atomic counter uses hardware atomic operations to achieve real-time updates of write operation counts, eliminating the synchronization overhead of traditional locking mechanisms. Through a layered locking strategy, these three components effectively isolate the locking scope of file metadata protection, task scheduling management, and state synchronization operations. This allows read operations to be executed concurrently without being blocked by write operations, background task scheduling to be unaffected by foreground writes, and write state updates to eliminate lock contention, thereby systematically reducing the probability of lock contention in high-concurrency environments.

[0061] As a specific implementation method, the solution in this embodiment is implemented as follows: The file tree read-write lock can be specifically implemented as a read-write lock based on pthread_rwlock_t. When multiple clients initiate file read requests simultaneously, the system allows multiple read threads to concurrently acquire the shared lock, while write requests require acquiring an exclusive lock; the task queue lock can use pthread_mutex_t mutex lock to protect the background task queue, ensuring the atomicity of enqueue and dequeue operations for COW copying tasks; the atomic counter can use the __sync_fetch_and_add function provided by GCC to atomically increment the write operation count, thereby quickly updating the Extent status information in scenarios with frequent writes.

[0062] Through the above solution, this embodiment effectively alleviates the lock contention problem in multi-user concurrent access scenarios, avoids performance bottlenecks caused by global lock contention in core operations such as Extent status checks and metadata updates, and significantly improves the response efficiency and throughput of the storage system under high-concurrency write loads.

[0063] In practical applications, some of the solutions described above in this embodiment propose a general method for storing large-size data. However, in this process, the system lacks cross-platform deployment capabilities and is difficult to run in different environments such as Linux and UEFI. This results in the storage solution being limited to a specific platform environment and unable to flexibly adapt to diverse deployment needs.

[0064] To address this, this embodiment further proposes to achieve cross-platform deployment through a platform adaptation layer, which includes at least one of the following: Linux platform adaptation layer, used to implement block device drivers and BIO forwarding; The UEFI platform adaptation layer is used to implement EFI_BLOCK_IO_PROTOCOL adaptation. The user-space adaptation layer is used to provide pthread / glibc environment support. Among them, the platform adaptation layer refers to an abstract interface layer, which can be implemented in various ways to isolate platform differences. Its purpose is to decouple the core logic of the storage system from the underlying platform details, allowing core storage methods to adapt to different operating environments without modification. The Linux platform adaptation layer can be understood as an adaptation module for the Linux operating system. It can be implemented by implementing block device drivers and BIO forwarding mechanisms, aiming to seamlessly connect to the Linux kernel's I / O subsystem and ensure efficient storage request processing at the operating system level. The UEFI platform adaptation layer is specifically an adaptation module for the UEFI firmware environment. It can be implemented by implementing EFI_BLOCK_IO_PROTOCOL, aiming to enable the storage system to directly access hardware devices in the early stages of system initialization, meeting the special deployment requirements of the firmware boot environment. The user-space adaptation layer can be understood as an adaptation module running in user space. It can be implemented by providing pthread thread libraries and glibc standard library support, aiming to enable the core storage logic to run independently in user space, avoiding dependence on kernel permissions and enhancing the flexibility of application layer deployment. Specifically, the solution in this embodiment uses a platform adaptation layer as a unified abstract interface between the core logic and the underlying platform. During data storage operations, the core storage method calls the standard interface of the platform adaptation layer to execute I / O requests. The platform adaptation layer dynamically selects the appropriate adaptation layer implementation based on the current operating environment. For example, in a Linux system, it calls the Linux platform adaptation layer to handle block device drivers and BIO forwarding; in a UEFI environment, it calls the UEFI platform adaptation layer to adapt firmware protocols; and in user space, it calls the user-mode adaptation layer to provide thread and library support. This achieves cross-platform compatibility without changing the core storage logic, ensuring the stable operation of the storage system in different environments. As a preferred embodiment, the specific implementation of this embodiment is as follows: When deployed to a Linux server environment, the platform adaptation layer loads the Linux platform adaptation layer, enabling the core storage logic to directly interact with the operating system I / O subsystem by implementing the Linux kernel's block device driver interface and BIO forwarding mechanism; when the system boots in a UEFI firmware environment, the platform adaptation layer switches to the UEFI platform adaptation layer, implementing EFI_BLOCK_IO_PROTOCOL to support hardware access during the system initialization phase; in user-space applications, the platform adaptation layer adopts a user-mode adaptation layer, providing pthread and glibc environment support, enabling the storage function to run independently in user mode without relying on kernel privileges. Through the above solution, this embodiment achieves seamless migration and stable operation of the storage method in Linux, UEFI and user-space environments, effectively solving the deployment limitation problem caused by platform differences, and enabling the storage solution to flexibly adapt to diverse deployment needs.

[0065] Furthermore, embodiments of this application also propose a computer-readable storage medium storing a program for general-purpose large-size data storage, wherein when the program for general-purpose large-size data storage is executed by a processor, it implements the steps of the method for general-purpose large-size data storage as described above.

[0066] Reference Figure 3 , Figure 3 This is a structural block diagram of the first embodiment of the general-purpose large-size data storage system of this application.

[0067] like Figure 3 As shown, the general-purpose large-size data storage system proposed in this application includes: Management module 10 is used to manage storage space in the core layer of the storage system by using fixed-size Extents as basic storage units, where each Extent has a Map state, a Free state, or a Parent state. Cow operation module 20 is used to perform copy-on-write (COW) operation when a data write request is received, including: checking the status of the target extent; if the target extent is in the parent state, allocating a new extent for the data and creating an asynchronous COW copy task to write the data to the newly allocated extent. The change module 30 is used to perform metadata change operations on the storage system. Consistency is ensured through a double-buffered log mechanism. The double-buffered log mechanism includes: first, appending the metadata changes to the memory log queue in the form of logs, and then periodically flushing the logs in memory to the disk in batches for persistence.

[0068] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0069] This embodiment effectively solves the problem of wasted space caused by immediately copying complete data blocks in large-size data storage by using fixed-size Extent to manage storage space, performing asynchronous copy-on-write operations, and implementing a double-buffered log mechanism. At the same time, it ensures the persistence reliability of metadata changes under abnormal conditions, and has the advantages of improving storage space utilization, enhancing data consistency assurance capabilities, and reducing write latency.

[0070] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0071] In addition, for technical details not described in detail in this embodiment, please refer to the general large-size data storage method provided in any embodiment of this application, which will not be repeated here.

[0072] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0073] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. The above are only preferred embodiments of this application and do not limit the patent scope of this application. All equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A general large-size data storage method, characterized in that, include: In the core layer of the storage system, fixed-size Extents are used as the basic storage units to manage the storage space. Each Extent has a Map state, a Free state, or a Parent state. When a data write request is received, a copy-on-write (COW) operation is performed, including: checking the status of the target extent; if the target extent is in the parent state, allocating a new extent for the data and creating an asynchronous COW copy task to write the data to the newly allocated extent. For metadata change operations in the storage system, consistency is guaranteed through a double-buffered log mechanism. The double-buffered log mechanism includes: first, appending metadata changes to the memory log queue in the form of logs, and then periodically flushing the logs in memory to the disk in batches for persistence.

2. The method according to claim 1, characterized in that, The method of managing storage space using fixed-size extents as basic storage units includes: The allocation status of all Extents is managed using a red-black tree; Calculate the corresponding Extent ID based on the file offset of the data write request; A proximity-first allocation strategy is used to allocate new extent space for the data.

3. The method according to claim 1, characterized in that, The state of the inspection target extent includes: If the target Extent is in a Free state, a new Extent is directly allocated, and the data is written to the newly allocated Extent, while the BIND log is recorded. If the target extent is in a Map state, the data is written directly to that extent.

4. The method according to claim 1, characterized in that, The creation of the asynchronous COW replication task includes: The COW replication task is placed in the background task queue. The COW replication task includes: copying data from the parent Extent to the newly allocated Extent, then writing the new data to the newly allocated Extent, and updating the Extent state to Map. The write request is returned immediately after the COW replication task is created, without waiting for the data replication to complete.

5. The method according to claim 1, characterized in that, The dual-buffered logging mechanism also includes log replay and organization: During system initialization, the memory state of the storage system is reconstructed by replaying the logs on the disk; Perform log cleanup regularly to compress the current system state into a minimal log sequence, thereby freeing up storage space.

6. The method according to claim 1, characterized in that, The method employs a multi-level locking mechanism for concurrency control, which includes: File tree read-write locks are used to protect file metadata and extent mappings, allowing multiple read operations to access concurrently; Task queue locks are used to manage background task scheduling; Atomic counters are used for write operation counting and state synchronization.

7. The method according to any one of claims 1 to 6, characterized in that, The method achieves cross-platform deployment through a platform adaptation layer, which includes at least one of the following: Linux platform adaptation layer, used to implement block device drivers and BIO forwarding; The UEFI platform adaptation layer is used to implement EFI_BLOCK_IO_PROTOCOL adaptation. The user-space adaptation layer is used to provide pthread / glibc environment support.

8. A general-purpose large-size data storage system, characterized in that, include: The management module is used to manage storage space in the core layer of the storage system, using fixed-size Extents as the basic storage units. Each Extent has a Map state, a Free state, or a Parent state. The Cow operation module is used to perform copy-on-write (COW) operations when a data write request is received. This includes: checking the status of the target Extent; if the target Extent is in the Parent state, allocating a new Extent for the data and creating an asynchronous COW copy task to write the data to the newly allocated Extent. The change module is used to perform metadata change operations on the storage system. Consistency is ensured through a double-buffered log mechanism. The double-buffered log mechanism includes: first, appending the metadata changes to the memory log queue in the form of logs, and then periodically flushing the logs in memory to the disk in batches for persistence.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the processor, when executing computer instructions stored in the memory, performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images