Snapshot processing method and device, equipment, medium and program product

Abstract

The invention discloses a snapshot processing method and device, equipment, a medium and a program product in the technical field of computers. According to the method and the device, the preset block granularity and the file fragment size are associated and determined, so that the single merging operation of snapshot merging is matched with the fragment dimension of actual file storage, and the suitability of the snapshot merging task is improved; meanwhile, space recycling is carried out according to the preset block granularity, and recycling of the whole block space according to the file fragment size is facilitated; and through the merging progress and the recovery progress, the data consistency risk generated by the snapshot merging operation and the front-end read-write operation is solved, the real-time state of the snapshot merging and the space recovery can be accurately tracked, the problem node can be quickly positioned when the abnormality occurs, the merging and recovery operations form a unified execution reference, and the efficiency of the system is improved. And finally, the technical effects of shortening the overall time consumption of the snapshot merging task and improving the execution efficiency are achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a snapshot processing method, apparatus, device, medium, and program product. Background Technology

[0002] Currently, continuous snapshot merging operations in storage systems gradually consume storage space due to redundant data storage. For example, after merging snapshot A into snapshot B, the original snapshot A still occupies storage space, and snapshot A is also added to the merged snapshot B. As more snapshot merging operations continue, storage space consumption will increase significantly. Furthermore, in multi-client environments, simultaneous snapshot merging operations and front-end read / write operations can pose certain data consistency risks.

[0003] Therefore, how to address the space consumption and data consistency risks caused by snapshot merging is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a snapshot processing method, apparatus, device, medium, and program product to solve the space occupation and data consistency risks caused by snapshot merging. The specific solution is as follows:

[0005] Firstly, this application provides a snapshot processing method, including:

[0006] Obtain a snapshot merge task;

[0007] The snapshot merging task is processed according to a preset block granularity; the preset block granularity is determined based on the file fragment size;

[0008] During the snapshot merging task processing, space reclamation is performed according to the preset block granularity, and the merging progress and reclamation progress are recorded accordingly.

[0009] Optionally, the snapshot merging task is processed according to a preset block granularity, including:

[0010] The snapshot merge task is parsed to determine the source snapshot, destination snapshot, and snapshot merge type;

[0011] Data is read from the source snapshot according to the preset block granularity;

[0012] Write the read data into the destination snapshot according to the snapshot merge type.

[0013] Optionally, reading data from the source snapshot according to the preset block granularity includes:

[0014] Query the current bitmap of the source snapshot to confirm whether there is valid data in the source snapshot;

[0015] If valid data is confirmed to exist in the source snapshot, multiple threads are used to read data of the preset block granularity from the source snapshot in parallel.

[0016] Optionally, it also includes:

[0017] If the latency of the system storage service exceeds the preset latency limit, the number of concurrent reads of the source snapshot will be reduced.

[0018] If the latency of the system storage service is lower than the preset latency limit, then the concurrency of reading the source snapshot is increased.

[0019] Optionally, before processing the snapshot merging task according to a preset block granularity, the method further includes:

[0020] Add a mutex lock to the snapshot merging task.

[0021] Optionally, the merging progress is recorded, including:

[0022] Record the merge completion address corresponding to the merge operation of each preset block granularity; if the system recovers after a failure, continue processing the snapshot merge task according to the merge completion address;

[0023] And / or, after each merge operation of a preset block granularity size, record the first bitmap update information of the source snapshot corresponding to the snapshot merge task and the second bitmap update information of the destination snapshot corresponding to the snapshot merge task, so that the client updates the local bitmap cache information according to the first bitmap update information and the second bitmap update information.

[0024] Optionally, the recycling progress is recorded, including:

[0025] After each space reclamation operation of a preset block size, record the location information of the reclaimed intervals;

[0026] Based on the interval location information, the response of the client to the access request initiated for the reclaimed interval is set to a preset error code, and the access request is redirected.

[0027] Secondly, this application provides a snapshot processing apparatus, comprising:

[0028] The acquisition module is used to acquire snapshot merge tasks;

[0029] The merging module is used to process the snapshot merging task according to a preset block granularity; the preset block granularity is determined based on the file fragment size.

[0030] The processing module is used to perform space reclamation according to the preset block granularity during the snapshot merging task processing, and record the merging progress and reclamation progress accordingly.

[0031] Optionally, the merging module is used for:

[0032] The snapshot merge task is parsed to determine the source snapshot, destination snapshot, and snapshot merge type;

[0033] Data is read from the source snapshot according to the preset block granularity;

[0034] Write the read data into the destination snapshot according to the snapshot merge type.

[0035] Optionally, the merging module is used for:

[0036] Query the current bitmap of the source snapshot to confirm whether there is valid data in the source snapshot;

[0037] If valid data is confirmed to exist in the source snapshot, multiple threads are used to read data of the preset block granularity from the source snapshot in parallel.

[0038] Optionally, it also includes:

[0039] The balancing module is used to reduce the concurrency of reading the source snapshot if the latency of the system storage service exceeds a preset latency limit, and to increase the concurrency of reading the source snapshot if the latency of the system storage service is lower than the preset latency limit.

[0040] Optionally, it also includes:

[0041] The locking module is used to add a mutex lock to the snapshot merging task before processing it according to a preset block granularity.

[0042] Optionally, the processing module is used for:

[0043] Record the merge completion address corresponding to the merge operation of each preset block granularity; if the system recovers after a failure, continue processing the snapshot merge task according to the merge completion address;

[0044] And / or, after each merge operation of a preset block granularity size, record the first bitmap update information of the source snapshot corresponding to the snapshot merge task and the second bitmap update information of the destination snapshot corresponding to the snapshot merge task, so that the client updates the local bitmap cache information according to the first bitmap update information and the second bitmap update information.

[0045] Optionally, the processing module is used for:

[0046] After each space reclamation operation of a preset block size, record the location information of the reclaimed intervals;

[0047] Based on the interval location information, the response of the client to the access request initiated for the reclaimed interval is set to a preset error code, and the access request is redirected.

[0048] Thirdly, this application provides an electronic device, comprising:

[0049] Memory, used to store computer programs;

[0050] A processor is configured to execute the computer program to implement the aforementioned disclosed snapshot processing method.

[0051] Fourthly, this application provides a readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned snapshot processing method.

[0052] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed snapshot processing method.

[0053] As can be seen from the above scheme, this application provides a snapshot processing method, including: obtaining a snapshot merging task; processing the snapshot merging task according to a preset block granularity; the preset block granularity is determined based on the file fragment size; during the snapshot merging task processing, space reclamation is performed according to the preset block granularity, and the merging progress and reclamation progress are recorded accordingly.

[0054] As can be seen, the technical effects of this application include:

[0055] (1) By associating the preset block granularity with the file fragment size, the individual merge operation of snapshot merging is made consistent with the actual file storage fragment dimension, avoiding invalid processing or excessive splitting caused by the mismatch between block granularity and file fragments, ensuring the synergy between the merge logic and the underlying storage architecture, and optimizing the rationality of task execution from a fundamental perspective. This allows for precise matching of file storage characteristics and improves the adaptability of snapshot merging tasks.

[0056] (2) During the snapshot merging task processing, space reclamation operations are performed in units of preset block granularity. This allows for the timely release of redundant storage space for merged block-level data, avoiding long-term resource occupation caused by batch reclamation after the overall task is completed. This achieves dynamic and efficient release of storage resources, significantly improving the turnover efficiency and utilization accuracy of storage space. Furthermore, space reclamation in units of preset block granularity facilitates the reclamation of the entire block of space based on the file fragment size, avoiding the need for hole-digging in file fragments. Thus, it can achieve both refined space reclamation and improved storage resource utilization.

[0057] (3) By recording the merging progress and the recycling progress separately, the real-time status of snapshot merging and space recycling can be accurately tracked, and the completion status of each stage of the task can be clearly presented. This not only provides clear data support for system monitoring, but also facilitates the quick location of problem nodes when anomalies occur, improves the controllability of task execution and operation and maintenance efficiency, and ensures the stable progress of the snapshot merging process.

[0058] (4) By monitoring the merge and reclamation progress, the data consistency risks arising from snapshot merging operations and front-end read / write operations can be mitigated. For example, after receiving a client access request, the reclamation progress is used to confirm whether there is valid data at the access address of the request (i.e., whether the access address has been reclaimed); the merge progress is used to confirm whether the data to be accessed by the request is at an address that has already been merged (i.e., the updated destination snapshot address), at an address that is about to be merged (i.e., the source snapshot address), or at the destination snapshot address before the merge. This ensures that the client accesses the latest data and completes the access request normally.

[0059] (4) By using a standardized processing mode with preset block granularity, the merging and recycling operations form a unified execution benchmark, reducing process interruptions or repetitive operations caused by inconsistent processing units. At the same time, refined progress management can avoid invalid waiting or resource waste, ultimately achieving the technical effect of shortening the overall time consumption of snapshot merging tasks and improving execution efficiency.

[0060] Correspondingly, the snapshot processing apparatus, device, readable storage medium, and computer program product provided in this application also have the above-mentioned technical effects. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0062] Figure 1 This is a flowchart of a snapshot processing method disclosed in this application;

[0063] Figure 2 This is a schematic diagram of a snapshot merging disclosed in this application;

[0064] Figure 3 This is a schematic diagram of a snapshot merging and space reclamation process disclosed in this application;

[0065] Figure 4 This is a schematic diagram of a lease mechanism disclosed in this application;

[0066] Figure 5 This is a schematic diagram of a fault scenario disclosed in this application;

[0067] Figure 6 This is a schematic diagram of another electronic device disclosed in this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] Currently, continuous snapshot merging operations in storage systems gradually consume storage space due to redundant data storage. For example, after merging snapshot A into snapshot B, the original snapshot A still occupies storage space, and snapshot A is also added to the merged snapshot B. As more snapshot merging operations continue, storage space consumption will increase significantly. Furthermore, in multi-client environments, simultaneous snapshot merging operations and front-end read / write operations can pose certain data consistency risks. Therefore, this application provides a snapshot processing solution that can address the space consumption and data consistency risks associated with snapshot merging.

[0070] See Figure 1 As shown in the figure, this application discloses a snapshot processing method, including:

[0071] S101, Obtain snapshot merge task.

[0072] In this embodiment, the snapshot merging task may include: a snapshot ID list, original file information, and the source and destination of the snapshot merging (i.e., the data copy direction). Specifically, a snapshot is a copy of the original file, which can be a disk file of each virtual machine in the storage system. Furthermore, the snapshot merging task may also include the size of each snapshot in the task, the merge type, the snapshot file handle of the snapshot ID list, and the metadata information of each snapshot (e.g., views, bitmaps, snapshot chains, etc.). Each snapshot corresponds to a bitmap, which records whether the data in the corresponding snapshot is valid. The snapshots can form a view or a snapshot chain, which can represent the relationship between different snapshots.

[0073] It should be noted that the merging of snapshots can be referenced in Table 1 and... Figure 2 .exist Figure 2 In this context, base is the original file; cur is the latest snapshot file; snap1 is the snapshot file at time snap1_t; and snap2 is the snapshot file at time snap2_t. Figure 2Each line in the text illustrates a comparison before and after a merge operation, such as... Figure 2 The first line shows: after merging snap2 into snap1, a snap_merged file is obtained; after the merge, this snap_merged file replaces the original snap1 and becomes the snapshot file at time snap1_t; and the snapshot file snap2 at time snap2_t and at that time are deleted, with the snapshot time directly changing from time snap1_t to time snap3; while time snap3 and its corresponding cur file remain unchanged before and after this merge. Accordingly, Figure 2 Other merging operations and comparisons before and after merging can be understood by referring to the foregoing content, and will not be repeated here in this embodiment.

[0074] Table 1

[0075]

[0076] S102. Process the snapshot merging task according to the preset block granularity; the preset block granularity is determined based on the file fragment size.

[0077] It should be noted that the file fragment size can specifically be: the fragment size of the virtual machine's disk file, which can be 512MB. Correspondingly, the preset block granularity can be an integer multiple of the file fragment size, such as: 512MB, 512MB×2, 512MB×5, etc.

[0078] In one implementation, processing the snapshot merging task according to a preset block granularity includes: parsing the snapshot merging task to determine the source snapshot, the destination snapshot, and the snapshot merging type; reading data from the source snapshot according to the preset block granularity; and writing the read data into the destination snapshot according to the snapshot merging type. Reading data from the source snapshot according to the preset block granularity includes: querying the current bitmap of the source snapshot to confirm whether valid data exists in the source snapshot; if valid data is confirmed, then using multiple threads to read data of the preset block granularity size from the source snapshot in parallel. Each thread can read data in a set amount each time, such as reading 128KB of data each time. After reading a total of one preset block granularity of data, a snapshot merging operation (i.e., writing to the destination snapshot) is initiated. This method queries the current bitmap of the source snapshot and confirms the existence of valid data, directly filtering out source snapshot processing scenarios without valid data. It eliminates redundant operations such as reading and transmitting empty or invalid data, reducing the unnecessary consumption of system resources (computing, storage, and network) from the source and improving the accuracy of task execution and resource utilization efficiency. Furthermore, parallel data reading improves task processing efficiency: after confirming the existence of valid data, multi-threaded parallel reading of data at a preset block granularity is used, breaking through the performance bottleneck of single-threaded serial reading and significantly shortening the time spent in the data reading stage. Simultaneously, the standardized data unit design with preset block granularity ensures a more balanced allocation of multi-threaded tasks, avoiding efficiency losses caused by uneven load between threads, and achieving high-speed, large-scale processing of source snapshot data.

[0079] It should also be noted that confirming valid data based on bitmaps ensures that all target data read by multiple threads is valid, avoiding task exceptions or result errors caused by reading invalid data; the unified data unit division with preset block granularity gives multi-threaded reading operations clear boundaries and standards, facilitating the orderly conduct of subsequent processes such as data splicing and verification, improving the stability of the overall data processing chain, and ensuring the reliability and orderliness of data reading.

[0080] S103. During the snapshot merging task processing, space reclamation is performed according to the preset block granularity, and the merging progress and reclamation progress are recorded accordingly.

[0081] This embodiment records the merge progress and recycling progress, which can provide assurance for fault recovery and data consistency. In one implementation, recording the merge progress includes: recording the merge completion address corresponding to each merge operation of a preset block granularity; if the system recovers after a fault, the snapshot merge task continues to be processed according to the merge completion address; and / or, after each merge operation of a preset block granularity, recording the first bitmap update information of the source snapshot corresponding to the snapshot merge task and the second bitmap update information of the destination snapshot corresponding to the snapshot merge task, so that the client updates its local bitmap cache information according to the first bitmap update information and the second bitmap update information.

[0082] As can be seen, this embodiment records the merge completion address corresponding to each preset block-level merge operation, providing a clear location basis for task continuation after system fault recovery. This avoids re-executing tasks from the beginning after a fault, significantly reducing resource consumption and time costs caused by repeated operations. It ensures that the snapshot merge process can accurately resume incomplete steps after abnormal interruption, improving the fault resistance and overall continuity of task execution, and realizing breakpoint resumption of snapshot merge tasks. This further enables more precise location of task breakpoints and improves fault recovery efficiency: the merge completion address, as a fine-grained breakpoint identifier at the preset block level, can accurately point to the last merged block position before the fault occurred. During system recovery, it is not necessary to traverse or verify all data blocks to quickly determine the continuation starting point, significantly shortening the task restart time after fault recovery and optimizing the rapid recovery capability after system faults.

[0083] Of course, the breakpoint resume mechanism based on the merged address can only perform subsequent operations on the blocks that have not been merged, avoiding the repeated processing of merged blocks. This can further reduce storage wear and computing resource waste caused by repeated data read and write, while reducing data consistency problems that may be caused by repeated operations, ensuring the accuracy and reliability of snapshot merge results, and reducing the risk of data redundancy and repeated processing.

[0084] It should also be noted that, after each preset block-level granularity merging operation, this embodiment specifically records the first bitmap update information of the source snapshot and the second bitmap update information of the destination snapshot. This ensures the real-time, completeness, and accuracy of bitmap change data, providing accurate and reliable data for client cache synchronization. It avoids caching errors caused by missing or delayed bitmap information, achieving fine-grained recording of bitmap update information during snapshot merging. This further ensures the consistency between the client's local bitmap cache and the snapshot merging state: the client can update its locally cached bitmap data promptly based on the recorded first and second bitmap update information, keeping the cached content synchronized with the actual bitmap state after snapshot merging. This effectively avoids snapshot data access errors caused by cache expiration or inconsistency, improving the accuracy and reliability of client data reading. Of course, through precise bitmap update information recording and client cache synchronization mechanisms, it reduces invalid operations such as repeated queries and data retrieval from the source caused by bitmap cache errors, reducing system interaction overhead and resource consumption. Simultaneously, it ensures the smoothness of the snapshot data access process, improving the overall system efficiency and stability in processing snapshot data, and optimizing the overall efficiency and stability of snapshot data access.

[0085] This embodiment records the reclamation progress, which can solve the data consistency problems between clients and between clients and the storage system that may be caused by snapshot merging. In one implementation, recording the reclamation progress includes: after each space reclamation operation of a preset block granularity size, recording the location information of the reclaimed interval; based on the interval location information, setting the response of the client to the access request initiated for the reclaimed interval to a preset error code, and performing a redirection operation on the access request.

[0086] As can be seen, this embodiment, by recording the reclamation progress, can accurately track the real-time status of space reclamation, clearly define the reclamation and non-reclaimed data ranges, and avoid data access conflicts or inconsistencies between clients and between clients and the storage system caused by the opaque data status during snapshot merging. This ensures the uniformity and accuracy of the data status of the entire storage system and effectively solves the data consistency problem caused by snapshot merging. Furthermore, this embodiment records the location information of the reclaimed range after each preset block-level space reclamation operation, allowing the system to quickly identify client access requests to the reclaimed range based on this information. By returning preset error codes and performing redirection operations, it prevents clients from accessing invalid reclaimed data and guides requests to the correct data location, improving the accuracy and effectiveness of access responses. This precisely controls access requests to the reclaimed range. This further improves the reliability of system access request processing: implementing error code return and redirection strategies for access requests to the reclaimed range effectively avoids system anomalies or data read errors caused by invalid data access, reduces the consumption of system resources by erroneous requests, ensures a good client access experience, and enhances the service stability of the storage system during snapshot merging and space reclamation.

[0087] It should also be noted that this embodiment can achieve a balance between the merging task and the front-end system storage services. Specifically, if the latency of the system storage services exceeds a preset latency limit, the concurrency of reading the source snapshot is reduced; if the latency of the system storage services is lower than the preset latency limit, the concurrency of reading the source snapshot is increased. Thus, by monitoring the latency of the system storage services, reducing the concurrency of reading the source snapshot when the latency exceeds the preset limit, and increasing the concurrency when the latency is lower than the preset limit, dynamic adaptation between the resource consumption of the merging task and the service quality of the front-end services is achieved. This avoids excessive resource consumption by the merging task leading to front-end service lag, ensuring the stability and smoothness of the overall system service.

[0088] Of course, to avoid write disorder caused by concurrent merging tasks, a mutex lock is added to the snapshot merging task before processing it according to the preset block granularity. This ensures that only a single merging task performs write operations at the same time, preventing data overwriting, disordered order, and other anomalies when multiple tasks write data in parallel. This ensures the atomicity and consistency of data writing during the snapshot merging process and improves the accuracy and reliability of the merging results.

[0089] This embodiment can also add exclusive leases to snapshot files in the snapshot merge task, provided that the file currently has no other leases. Once the exclusive lease takes effect, it will prevent the creation of other leases, and file read and modification operations are supported during the lease period. If a lease preemption event occurs in the storage system (such as creating a new snapshot, virtual machine migration, etc.), the snapshot merge task will be terminated immediately, and all clients will be notified to perform adaptive cache updates to ensure data consistency.

[0090] As can be seen, this embodiment determines the snapshot merging operation by associating the preset block granularity with the file fragment size, making the individual merging operation of snapshot merging match the actual file storage fragment dimension, thus improving the adaptability of the snapshot merging task. At the same time, space reclamation with the preset block granularity is beneficial for reclamation of the entire space based on the file fragment size. Furthermore, by using the merging progress and reclamation progress, it not only solves the data consistency risk caused by snapshot merging operations and front-end read / write operations, but also accurately tracks the real-time status of snapshot merging and space reclamation, facilitating rapid location of problem nodes when anomalies occur. It also makes the merging and reclamation operations form a unified execution benchmark, ultimately achieving the technical effect of shortening the overall time consumption and improving the execution efficiency of the snapshot merging task.

[0091] Please see Figure 3 A snapshot merging and space reclamation process includes:

[0092] 1. Create a snapshot merge task.

[0093] The snapshot merge task can include a list of snapshot IDs that need to be merged, image file information, and a determination of the source and destination of the snapshot merge (i.e., the direction of data copying), which are used as unique identifiers for subsequent merge tasks.

[0094] 2. Add a distributed mutex lock to the snapshot file of the task to ensure that there is only one merge task for the same image, thus ensuring mutual exclusion and preventing multiple tasks from running concurrently, which could lead to abnormal merged data and achieve concurrency control.

[0095] 3. Perform snapshot merge pre-processing.

[0096] The pre-merge operations include: loading metadata, opening handles to the snapshots to be merged, and obtaining image sizes. More specifically, this may involve obtaining snapshot file handles from the snapshot ID list, snapshot metadata information, the size of each snapshot file in the merge task, and the merge type, which will be used for data copying during the subsequent snapshot merge process.

[0097] 4. Speed ​​up the lease process so that client operations can be detected while merging tasks are being processed.

[0098] Specifically, during the snapshot merging process, the system promptly detects any new snapshots or other operations on the snapshot chain, and then notifies the snapshot merging task to take corresponding actions, such as terminating the snapshot merging or updating metadata information.

[0099] 5. Perform data copying for snapshot merging tasks at the block level.

[0100] Specifically, snapshot merging is performed according to a fixed interval (e.g., 512MB). For example, the snapshot bitmap of the merging source is queried. If the snapshot contains valid data, 8 threads are used to read the source snapshot in 128KB blocks. Then, after accumulating a block size (512MB), the target snapshot is written. The system can also detect storage service latency. If the latency is too high, the number of threads and the merge depth (concurrent read count) are adjusted to reduce storage service latency. Once the storage service latency is restored, the number of threads and the merge depth can be adjusted back.

[0101] 6. Set merge breakpoints.

[0102] Specifically, every 512MB is merged, the merge position (the address where the merge is complete, i.e., the address where the merge is complete in the latest destination snapshot) is recorded. Therefore, in fault scenarios, merging can resume from the merge position where the last fault occurred, achieving the effect of resuming interrupted transmission.

[0103] 7. Adaptively update the bitmaps of the source and destination snapshots.

[0104] Specifically, after performing a merge operation, the bitmap information corresponding to the snapshot is cleared, such as marking the data at the corresponding position in the bitmap as invalid, so that subsequent read operations cannot index this position.

[0105] 8. Unmap (reclaim space) the space occupied by the snapshot.

[0106] Specifically, sending a discard operation to reclaim space can be achieved by converting the discard semantics into a file deletion operation based on the file fragment information, thus achieving the effect of space reclamation; that is, reclaiming space in 512MB increments; the amount of data deleted by each deletion operation is relatively consistent with the fragment size, which can avoid deleting fragments by creating holes.

[0107] 9. Post-snapshot merge operation.

[0108] The post-snapshot merging operations include: clearing task information and snapshot space usage information (the space occupied by the snapshot itself) and displaying the corresponding content.

[0109] 10. Snapshot merge task completed.

[0110] It should be noted that the above process is applicable to all snapshot merging types, including CUR2BASE, SNAP2SNAP, SNAP2CUR, and multi-in-one scenarios (merging multiple deleted snapshots into one snapshot). This solution performs the merge at a fixed block granularity (e.g., 512MB) and releases space at the block granularity during the snapshot merging process, thus solving the space occupation problem during snapshot merging.

[0111] It should be noted that, to ensure data consistency and business performance stability during the snapshot merging process, strong consistency in snapshot metadata updates and metadata access across all platforms is required. Snapshot metadata operations involve snapshot bitmaps, snapshot views, and the invalidation period during the snapshot merging process; there are also various metadata update orders, update timeframes, bitmap invalidation periods, and snapshot metadata memory invalidation. Therefore, this embodiment again uses a lease mechanism to address this consistency issue, ensuring the correctness of the snapshot view. Please refer to [link to relevant documentation]. Figure 4 Similar to write locks, snapshot merge tasks require exclusive leases on the files they operate on, meaning the files must currently have no other leases. Once an exclusive lease is added, no other leases can be added. While using an exclusive lease, the file can be read and modified. Therefore, if a lease preemption event occurs in the storage system (such as creating a new snapshot, virtual machine migration, etc.), the merge task terminates, and all clients are notified to perform adaptive cache updates.

[0112] Please refer to Table 2. For the failure intervals and bitmap failure intervals generated by snapshot merging, record the attributes according to Table 2 to record the merging progress and reclamation progress. Specifically, a three-replica mechanism can be used to ensure strong consistency of extended attributes, and based on each extended attribute, ensure the consistency of metadata under failure scenarios.

[0113] Table 2

[0114]

[0115] Combination Figure 3 The process shown Figure 4 The lease mechanism shown and the extended attributes shown in Table 2 can be used for... Figure 5 The illustrations demonstrate how to resolve data inconsistency scenarios in various data failure scenarios. It is evident that the extended attributes and lease mechanism provided in this case can resolve most data inconsistency scenarios. Specifically, these include: (1) failure to send control plane tasks, failure to send data plane requests, and failure to send requests to merge target hosts. (2) failure to update snapshot bitmap operations on the data plane, bitmap update failures, and in multi-in-one scenarios, only some snapshot bitmap updates succeed. (3) failure to set the scoff xattr for the merge source of the snapshot bitmap, failure to set the scoff xattr for the merge target, etc. (4) failure to update the cache of the snapshot view. (5) failure to set the recycled xattr for the global sharding module, etc. The storage service side refers to the system storage side, including storage disks.

[0116] In one example, a centralized metadata center and a data-separated architecture can be adopted to redesign the snapshot management organization. Snapshot space release during the snapshot merging process is uniformly managed by the metadata center. After a snapshot is deleted, only the invalidation period is recorded to the metadata center, and data space release can be performed using asynchronous garbage collection. Simultaneously, distributed lock semantics are implemented to ensure data consistency across multiple clients. This improves storage space utilization and also solves the multi-client data consistency problem through coarse-grained (file-level based on snapshot leases) and fine-grained (file-level based on file extended attributes) snapshot metadata.

[0117] The following describes a snapshot processing apparatus provided in an embodiment of this application. The snapshot processing apparatus described below and the snapshot processing method described above can be referred to each other.

[0118] This application discloses a snapshot processing apparatus, including:

[0119] The acquisition module is used to acquire snapshot merge tasks;

[0120] The merging module is used to process the snapshot merging task according to a preset block granularity; the preset block granularity is determined based on the file fragment size.

[0121] The processing module is used to perform space reclamation according to the preset block granularity during the snapshot merging task processing, and record the merging progress and reclamation progress accordingly.

[0122] In one implementation, the merging module is used to:

[0123] The snapshot merge task is parsed to determine the source snapshot, destination snapshot, and snapshot merge type;

[0124] Data is read from the source snapshot according to the preset block granularity;

[0125] Write the read data into the destination snapshot according to the snapshot merge type.

[0126] In one implementation, the merging module is used to:

[0127] Query the current bitmap of the source snapshot to confirm whether there is valid data in the source snapshot;

[0128] If valid data is confirmed to exist in the source snapshot, multiple threads are used to read data of the preset block granularity from the source snapshot in parallel.

[0129] In one implementation, it further includes:

[0130] The balancing module is used to reduce the concurrency of reading the source snapshot if the latency of the system storage service exceeds a preset latency limit, and to increase the concurrency of reading the source snapshot if the latency of the system storage service is lower than the preset latency limit.

[0131] In one implementation, it further includes:

[0132] The locking module is used to add a mutex lock to the snapshot merging task before processing it according to a preset block granularity.

[0133] In one implementation, the processing module is used to:

[0134] Record the merge completion address corresponding to the merge operation of each preset block granularity; if the system recovers after a failure, continue processing the snapshot merge task according to the merge completion address;

[0135] And / or, after each merge operation of a preset block granularity size, record the first bitmap update information of the source snapshot corresponding to the snapshot merge task and the second bitmap update information of the destination snapshot corresponding to the snapshot merge task, so that the client updates the local bitmap cache information according to the first bitmap update information and the second bitmap update information.

[0136] In one implementation, the processing module is used to:

[0137] After each space reclamation operation of a preset block size, record the location information of the reclaimed intervals;

[0138] Based on the interval location information, the response of the client to the access request initiated for the reclaimed interval is set to a preset error code, and the access request is redirected.

[0139] As can be seen, this embodiment provides a snapshot processing device that can solve the space occupation and data consistency risks caused by snapshot merging.

[0140] The following describes an electronic device provided by an embodiment of this application. The electronic device described below can be referred to in conjunction with the snapshot processing method and apparatus described above.

[0141] This application discloses an electronic device, including:

[0142] Memory, used to store computer programs;

[0143] A processor is configured to execute the computer program to implement the methods disclosed in any of the above embodiments.

[0144] Please refer to Figure 6 , Figure 6This is a schematic diagram of another electronic device provided in this embodiment. This electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the snapshot processing device. Furthermore, the central processing unit 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the electronic device 301.

[0145] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341. Examples include Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0146] exist Figure 6 In this context, application 342 may be a program that executes the snapshot processing method, and data 344 may be data required or generated for executing the snapshot processing method.

[0147] The steps in the snapshot processing method described above can be implemented by the structure of an electronic device.

[0148] The following describes a readable storage medium provided in an embodiment of this application. The readable storage medium described below can be referred to in conjunction with the snapshot processing method, apparatus and device described above.

[0149] A readable storage medium is provided for storing a computer program, wherein the computer program, when executed by a processor, implements the snapshot processing method disclosed in the foregoing embodiments. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0150] The following describes a computer program product provided by an embodiment of this application. The computer program product described below can be referred to in conjunction with other embodiments described herein.

[0151] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed snapshot processing method.

[0152] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the steps in any of the above embodiments.

[0153] The terms “first,” “second,” “third,” “fourth,” etc., used in this application (if applicable) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, or apparatus.

[0154] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0155] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0156] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of readable storage medium known in the art.

[0157] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A snapshot processing method, characterized in that, include: Obtain a snapshot merge task; The snapshot merging task is processed according to a preset block granularity; The preset block granularity is determined based on the file fragment size; During the snapshot merging task processing, space reclamation is performed according to the preset block granularity, and the merging progress and reclamation progress are recorded accordingly.

2. The snapshot processing method according to claim 1, characterized in that, The snapshot merging task is processed according to a preset block granularity, including: The snapshot merge task is parsed to determine the source snapshot, destination snapshot, and snapshot merge type; Data is read from the source snapshot according to the preset block granularity; Write the read data into the destination snapshot according to the snapshot merge type.

3. The snapshot processing method according to claim 2, characterized in that, Reading data from the source snapshot according to the preset block granularity includes: Query the current bitmap of the source snapshot to confirm whether there is valid data in the source snapshot; If valid data is confirmed to exist in the source snapshot, multiple threads are used to read data of the preset block granularity from the source snapshot in parallel.

4. The snapshot processing method according to claim 3, characterized in that, Also includes: If the latency of the system storage service exceeds the preset latency limit, the number of concurrent reads of the source snapshot will be reduced. If the latency of the system storage service is lower than the preset latency limit, then the concurrency of reading the source snapshot is increased.

5. The snapshot processing method according to any one of claims 1 to 4, characterized in that, Record the merge progress, including: Record the merge completion address corresponding to the merge operation of each preset block granularity; if the system recovers after a failure, continue processing the snapshot merge task according to the merge completion address; And / or, after each merge operation of a preset block granularity size, record the first bitmap update information of the source snapshot corresponding to the snapshot merge task and the second bitmap update information of the destination snapshot corresponding to the snapshot merge task, so that the client updates the local bitmap cache information according to the first bitmap update information and the second bitmap update information.

6. The snapshot processing method according to any one of claims 1 to 4, characterized in that, Record the recycling progress, including: After each space reclamation operation of a preset block size, record the location information of the reclaimed intervals; Based on the interval location information, the response of the client to the access request initiated for the reclaimed interval is set to a preset error code, and the access request is redirected.

7. A snapshot processing apparatus, characterized in that, include: The acquisition module is used to acquire snapshot merge tasks; The merging module is used to process the snapshot merging task according to a preset block granularity; The preset block granularity is determined based on the file fragment size; The processing module is used to perform space reclamation according to the preset block granularity during the snapshot merging task processing, and record the merging progress and reclamation progress accordingly.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images