Snapshot processing method, apparatus and electronic equipment based on object bucket

Abstract

This application discloses a snapshot processing method, apparatus, and electronic device based on object storage buckets. The method includes: responding to a snapshot comparison command via an object storage gateway to determine a first snapshot and a second snapshot corresponding to the object storage bucket; obtaining a target log based on the generation time of the first snapshot and the generation time of the second snapshot, wherein the target log is stored in a preset area and consists of object operation records sorted according to the completion time of modification operations; determining a first object difference set between the first snapshot and the second snapshot based on the target log, wherein the first object difference set includes objects that differ between the second snapshot and the first snapshot, and the objects are data stored in the object storage bucket; and performing incremental backup of the data generated by the object storage bucket during a target time period based on the first object difference set. This application solves the technical problem of low backup efficiency in existing technologies for backing up object storage buckets.

Description

Technical Field

[0001] This application relates to the field of data storage, and more specifically, to a snapshot processing method, apparatus, and electronic device based on object storage buckets. Background Technology

[0002] With the advent of the big data era, the explosive growth of data has brought severe challenges to traditional storage. Distributed storage has emerged to address this challenge, providing users with massive, highly reliable, highly scalable, and decentralized storage services. However, in massive storage scenarios, distributed storage faces numerous challenges, particularly in the storage of unstructured data, where data security and backup are critical issues.

[0003] Currently, in the field of object storage, data backup solutions are usually multi-version control solutions. Under this solution, a new version of the object is generated after each modification operation. Multiple versions of the object are stored in an object storage bucket at the same time. When a backup is needed, a specific version of the object can be specified, and the corresponding data of the object can be retrieved and backed up.

[0004] However, since the entire object bucket needs to be backed up at the same time during backup, and the existing solutions do not record the time when the object is modified, each time the object bucket is backed up, all versions of the object can only be backed up, i.e., a full backup, which leads to low backup efficiency.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a snapshot processing method, apparatus, and electronic device based on object storage buckets, to at least solve the technical problem of low backup efficiency in the prior art when backing up object storage buckets.

[0007] According to one aspect of the embodiments of this application, a snapshot processing method based on an object storage bucket is provided, comprising: responding to a snapshot comparison instruction through an object storage gateway to determine a first snapshot and a second snapshot corresponding to the object storage bucket, wherein the generation time of the first snapshot is earlier than the generation time of the second snapshot, and the snapshot comparison instruction is used to compare two snapshots with different generation times; obtaining a target log based on the generation time of the first snapshot and the generation time of the second snapshot, wherein the target log is stored in a preset area, and the target log is an object operation record sorted according to the completion time of the modification operation; determining a first object difference set between the first snapshot and the second snapshot based on the target log, wherein the first object difference set includes objects that differ between the second snapshot and the first snapshot, and the objects are data stored in the object storage bucket; and performing incremental backup of the data generated by the object storage bucket in a target time period based on the first object difference set, wherein the target time period is the time interval between the generation time of the first snapshot and the generation time of the second snapshot.

[0008] Furthermore, the snapshot processing method based on object storage buckets also includes: responding to a snapshot creation command to obtain a pre-snapshot, wherein the snapshot field of the pre-snapshot stores object metadata that has not been snapshotted; persisting the pre-snapshot to generate a new snapshot, wherein the new snapshot includes at least a snapshot number and a unique snapshot identifier corresponding to the snapshot number, wherein the snapshot number of different snapshots is positively correlated with the generation time of each snapshot, and the unique snapshot identifier is a string whose byte order decreases as the snapshot number increases.

[0009] Furthermore, the snapshot processing method based on object buckets also includes: responding to a snapshot deletion command, determining the third snapshot and the fourth snapshot corresponding to the object bucket, wherein the third snapshot is the snapshot to be deleted, and the fourth snapshot is the snapshot following the third snapshot; determining a second object difference set based on the target log, wherein the objects in the second object difference set are first objects, and the first objects are used to characterize objects that have been modified in the fourth snapshot compared to the third snapshot.

[0010] Furthermore, the snapshot processing method based on object storage buckets also includes: after determining the second object difference set based on the target log, determining the historical object metadata and historical user data corresponding to the first object; and deleting the historical object metadata, historical user data, and the third snapshot.

[0011] Furthermore, the snapshot processing method based on object storage bucket also includes: responding to a snapshot rollback command, determining all historical snapshots at the current time, wherein the snapshot rollback command includes at least a target snapshot number, and the historical snapshot corresponding to the target snapshot number is the target historical snapshot to which the snapshot is to be rolled back; deleting data in the pre-snapshot; and rolling back the current snapshot to the target historical snapshot.

[0012] Furthermore, the snapshot processing method based on object storage buckets also includes: after rolling back the current snapshot to the target historical snapshot, updating the snapshot search path, wherein the snapshot search path is used to characterize the association between multiple historical snapshots corresponding to the object storage bucket.

[0013] Furthermore, the snapshot processing method based on object storage buckets also includes: the object storage gateway includes at least a first object storage gateway and a second object storage gateway, and there is a time difference between the first object storage gateway and the second object storage gateway when receiving instructions; the time when the first object storage gateway generates the first snapshot is determined as the first time, and the time when the first object storage gateway generates the second snapshot is determined as the second time; the time when the second object storage gateway generates the first snapshot is determined as the third time, and the time when the second object storage gateway generates the second snapshot is determined as the fourth time; the time order between the first time and the third time is detected, and the time with the earlier time order is determined as the first target time; the time order between the second time and the fourth time is detected, and the time with the later time order is determined as the second target time; and a first object difference set between the first snapshot and the second snapshot is determined based on the first target time and the second target time.

[0014] According to another aspect of the embodiments of this application, a snapshot processing apparatus based on an object storage bucket is also provided, comprising: a first determining module, configured to determine a first snapshot and a second snapshot corresponding to the object storage bucket in response to a snapshot comparison instruction via an object storage gateway, wherein the generation time of the first snapshot is earlier than the generation time of the second snapshot, and the snapshot comparison instruction is used to compare two snapshots with different generation times; an acquiring module, configured to acquire a target log based on the generation time of the first snapshot and the generation time of the second snapshot, wherein the target log is stored in a preset area, and the target log is an object operation record sorted according to the completion time of the modification operation; a second determining module, configured to determine a first object difference set between the first snapshot and the second snapshot based on the target log, wherein the first object difference set includes objects that differ between the second snapshot and the first snapshot, and the objects are data stored in the object storage bucket; and a backup module, configured to perform incremental backup of the data generated by the object storage bucket in a target time period based on the first object difference set, wherein the target time period is the time interval between the generation time of the first snapshot and the generation time of the second snapshot.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program is configured to execute the above-described object bucket-based snapshot processing method at runtime.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are configured to run the programs, wherein the programs are configured to execute the above-described object bucket-based snapshot processing method at runtime.

[0017] In the technical solution of this application, a method is adopted to generate snapshots for object storage buckets and perform incremental backups of the object storage buckets based on the snapshot generation time. First, the object storage gateway responds to the snapshot comparison command to determine the first snapshot and the second snapshot corresponding to the object storage bucket. The generation time of the first snapshot is earlier than that of the second snapshot. The snapshot comparison command is used to compare the two snapshots with different generation times. Then, the target log is obtained based on the generation times of the first and second snapshots, and a first object difference set between the first and second snapshots is determined based on the target log. Finally, incremental backups are performed on the data generated by the object storage bucket during the target time period based on the first object difference set. The target log is stored in a preset area and consists of object operation records sorted according to the completion time of modification operations. The first object difference set includes objects that differ between the second and first snapshots, and these objects are the data stored in the object storage bucket. The target time period is the time interval between the generation time of the first snapshot and the generation time of the second snapshot.

[0018] As described above, since snapshots can be used to record the state information of object buckets, including object information within the bucket, this application can record the state information of object buckets at multiple different times by taking snapshots. Furthermore, since this application also records the snapshot generation time, and the target log records modification operations on objects and their timestamps, this application can compare the generation times of the first and second snapshots and the target log to identify the objects that have changed between the first and second snapshots, thus obtaining the first object difference set. It is noteworthy that since the first object difference set includes objects that differ between the second and first snapshots, when backing up the object bucket, incremental backup can be achieved simply by using the generation times of the two snapshots as a time reference and combining them with the objects in the first object difference set. Because the amount of data required for incremental backup is much smaller than that required for full backup, this application improves backup efficiency and reduces backup time by using incremental backup for object bucket backup.

[0019] Therefore, the technical solution of this application achieves the goal of backing up object storage buckets in an incremental backup manner, thereby improving the backup efficiency of object storage buckets and solving the technical problem of low backup efficiency in the prior art when backing up object storage buckets. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram of an optional multi-version storage object method based on existing technology;

[0022] Figure 2 This is a schematic diagram of another alternative method for storing multi-version objects according to existing technology;

[0023] Figure 3 This is a flowchart of an optional snapshot processing method based on an object bucket according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of an optional snapshot management system according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of an optional object snapshot copy according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of an optional snapshot comparison process according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of an optional snapshot creation process according to an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of an optional snapshot search link according to an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of an optional snapshot rollback process according to an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of an optional multi-object storage gateway snapshot comparison process according to an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of an optional snapshot processing apparatus according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] Example 1

[0035] According to an embodiment of this application, an embodiment of a snapshot processing method based on object buckets is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] Currently, in the field of object storage, data backup solutions typically employ multi-version control (MPC). Under this approach, each modification to an object generates a new version, and multiple versions are simultaneously stored in a single object storage bucket. Object modification operations include at least object upload and object deletion.

[0037] Optional, Figure 1 A schematic diagram of a method for storing multi-version objects according to the prior art is shown, such as... Figure 1 As shown, suppose there are two versions of object 1 stored in the object bucket, namely version A and version B. If the user performs a PUT operation (i.e., object upload operation) on object 1, a new version of object 1 will be generated in the object bucket.

[0038] in addition, Figure 2 A schematic diagram of another method for storing multi-version objects according to the prior art is shown, such as... Figure 2 As shown, assuming that there are two versions of object 1 stored in the object bucket, namely version A and version B, if the user performs a delete operation on object 1 (i.e., object deletion operation), a new version of object 1 will still be generated in the object bucket, such as version C of object 1.

[0039] As described above, existing technology stores modifications to an object as a single, different version of that object in an object bucket. When a backup is needed, a specific version of the object can be specified, and the corresponding data can be retrieved and backed up.

[0040] However, in existing technologies, each modification to an object generates a new version, consuming a significant portion of storage resources and causing storage shortages. Furthermore, as those skilled in the art know, backup services typically involve two crucial factors: granularity and time. Backup services usually require backing up the entire object bucket, but existing multi-version object storage methods back up objects at the object-by-object granularity, failing to meet the requirement of backing up the entire object bucket. Simultaneously, because existing solutions do not record the time of object modification operations, they cannot use time as a reference to obtain discrepancies and thus cannot intelligently perform a full backup of all objects, resulting in low backup efficiency.

[0041] To address the aforementioned problems, this application provides an embodiment of a snapshot processing method based on object buckets, wherein... Figure 3 This is a flowchart of an optional object bucket-based snapshot processing method according to an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps:

[0042] Step S301: The object storage gateway responds to the snapshot comparison command to determine the first snapshot and the second snapshot corresponding to the object storage bucket.

[0043] In step S301, the first snapshot was generated earlier than the second snapshot, and the snapshot comparison instruction is used to compare the two snapshots generated at different times.

[0044] Optionally, the aforementioned snapshot is used to record the state information of the object bucket at the time the snapshot was generated. This state information includes at least the object information within the object bucket. Both the first and second snapshots are snapshots of the same object bucket, but they were generated at different times. The first snapshot can be understood as the snapshot preceding the second snapshot, or it can be one of the n snapshots preceding the second snapshot.

[0045] Optionally, a snapshot processing system can be used as the execution entity of the snapshot processing method in the embodiments of this application, wherein, Figure 4 A schematic diagram of an optional snapshot processing system according to an embodiment of this application is shown, such as... Figure 4 As shown, the snapshot processing system includes a management platform, an object storage gateway, a metadata database, and a data pool. The management platform is used for deploying and managing the cluster; the object storage gateway receives various instructions for processing snapshots and executes related snapshot processing operations; the metadata database stores object metadata and target logs; and the data pool stores user data.

[0046] Furthermore, since metadata contains index information of user data content, the correct user data content can definitely be found through metadata. Therefore, in this application, snapshots only need to protect object metadata.

[0047] Optionally, the metadata database described above is a key-value database. The metadata database stores keys sorted in ascending byte order. The metadata database supports traversal based on character prefixes. Since the snapshot protects object metadata, a snapshot ID (snapshot identifier) ​​is added to the metadata key as the basis for sorting and differentiation.

[0048] The key design for object metadata is as follows:

[0049] ${ns}:${bid}:${obj_name}\x00${snap}\x00${version}\x00${suffix}

[0050] Parameter explanation:

[0051] -ns object type

[0052] -bid: bucket id

[0053] -obj_name object name

[0054] -\x00 separator, the smallest ASCII code character

[0055] -snap: Unique identifier for the snapshot

[0056] -version: multiple version numbers

[0057] In an optional embodiment, this application uses ROW (Redirect-On-Write) technology to take snapshots of object buckets, wherein, Figure 5 A schematic diagram of an optional object snapshot copy according to an embodiment of this application is shown, such as... Figure 5As shown, the same object can have multiple copies within different snapshot versions. For example, from the user's perspective, the current snapshot version of object objA is objA version2, the current snapshot version of object objB is objB version3, and the current snapshot version of object objC is objC version1. From the underlying perspective, the snapshot version copy of object objA is objA version1, and the snapshot version copies of object objB are objB version1 and objB version1.

[0058] It should be noted that since snapshots can be used to record the state information of object buckets, including object information in the object bucket, this application can record the state information of object buckets at multiple different times by taking snapshots of object buckets.

[0059] Step S302: Obtain the target log based on the generation time of the first snapshot and the generation time of the second snapshot.

[0060] In step S302, the target log is stored in a preset area. The target log is a record of object operations sorted according to the completion time of the modification operation.

[0061] Specifically, after obtaining the generation times of the first and second snapshots, the snapshot processing system can extract target logs based on these times. The target logs record all user modifications to objects, and these operation records are sorted according to the completion time of each modification. Furthermore, the target logs are stored in a preset area, which can be a metadata database.

[0062] In addition, object modification operations include at least object deletion and object upload operations. Object deletion can be understood as deleting part of the content of the object, while object upload can be understood as adding content to the object. In the object storage bucket, a new version of the object is generated every time a modification operation is performed on a certain object.

[0063] Step S303: Determine the first object difference set between the first snapshot and the second snapshot based on the target log.

[0064] In step S303, the first object difference set includes objects that differ from the first snapshot, and the objects are data stored in the object storage bucket.

[0065] Optionally, in snapshot use cases, a large part of the demand comes from backups, which are divided into full backups and incremental backups. Full backups are self-explanatory; they back up all the data in the entire object storage bucket. Incremental backups, on the other hand, refer to backing up only the data within a certain time period.

[0066] As described in step S301 above, the bucket snapshot method provided in this application uses the ROW (Return on Frame) approach. The data within a snapshot is exactly the data from the end point of the previous snapshot to the end point of the current snapshot (or the current time), which coincides with the requirements of incremental backup. Therefore, to achieve the requirements of incremental backup, the snapshot processing system needs to obtain the set of differences between snapshots.

[0067] like Figure 6 As shown, since the target log records the modification times of objects, the snapshot processing system can determine the chronological order of objects by obtaining the target log. Furthermore, based on the snapshot's creation time, the snapshot processing system can also determine the chronological order between different snapshots. Figure 6 As shown, any object must belong to a snapshot. Based on this, the snapshot processing system can obtain the difference objects between snapshots by recording the time point when the snapshot was created and obtaining all objects within this time point.

[0068] like Figure 6 As shown, assuming the first snapshot is Snapshot 1 and the second snapshot is Snapshot 2, with Snapshot 1 generated at time T1 and Snapshot 2 generated at time T2, the object bucket status information recorded in Snapshot 1 includes the batch upload of 100 million objects and the upload of obj1. The object bucket status information recorded in Snapshot 2 includes the upload of obj2, the upload of obj3, and the deletion of obj1. Furthermore, in... Figure 6 The image also shows snapshot 3, which records the status information of the object storage bucket as uploaded obj4.

[0069] Optionally, the formula for the first object difference set is diff(n, m), where n and m are snapshot IDs, such as snapshot 1 and snapshot 2. The difference object between two time points is determined as Tdiff(T1, T2), where T1 and T2 are time points. Then, according to... Figure 6 From snapshots 1 and 2, we can see that diff(snap1, snap2) = T, diff(T1, T2) = {put{obj2, obj3}, delete{obj1}2}.

[0070] Step S304: Perform incremental backup of the data generated by the object storage bucket during the target time period based on the first object difference set.

[0071] In step S304, the target time period is the time interval between the generation time of the first snapshot and the generation time of the second snapshot. For example, as... Figure 6 As shown, the target time period is the time period between T1 and T2.

[0072] It is important to note that since the first object difference set includes objects that differ from the first snapshot in the second snapshot, when backing up the object bucket, incremental backup of the object bucket can be achieved simply by using the generation times of the two snapshots as a time reference and combining them with the objects in the first object difference set. Because the amount of data required for incremental backup is significantly less than that required for full backup, this application's incremental backup method for object bucket backup improves backup efficiency and reduces backup time.

[0073] Based on the above steps S301 to S304, the technical solution of this application adopts a method of generating snapshots for object storage buckets and performing incremental backups of object storage buckets according to the snapshot generation time. First, the object storage gateway responds to the snapshot comparison command to determine the first snapshot and the second snapshot corresponding to the object storage bucket. The generation time of the first snapshot is earlier than the generation time of the second snapshot. The snapshot comparison command is used to compare two snapshots with different generation times. Then, the target log is obtained according to the generation time of the first snapshot and the generation time of the second snapshot, and the first object difference set between the first snapshot and the second snapshot is determined according to the target log. Finally, the data generated by the object storage bucket in the target time period is incrementally backed up according to the first object difference set. The target log is stored in a preset area and is an object operation record sorted according to the completion time of the modification operation. The first object difference set includes objects that differ between the second snapshot and the first snapshot. The objects are the data stored in the object storage bucket. The target time period is the time interval between the generation time of the first snapshot and the generation time of the second snapshot.

[0074] As described above, since snapshots can be used to record the state information of object buckets, including object information within the bucket, this application can record the state information of object buckets at multiple different times by taking snapshots. Furthermore, since this application also records the snapshot generation time, and the target log records modification operations on objects and their timestamps, this application can compare the generation times of the first and second snapshots and the target log to identify the objects that have changed between the first and second snapshots, thus obtaining the first object difference set. It is noteworthy that since the first object difference set includes objects that differ between the second and first snapshots, when backing up the object bucket, incremental backup can be achieved simply by using the generation times of the two snapshots as a time reference and combining them with the objects in the first object difference set. Because the amount of data required for incremental backup is much smaller than that required for full backup, this application improves backup efficiency and reduces backup time by using incremental backup for object bucket backup.

[0075] Therefore, the technical solution of this application achieves the goal of backing up object storage buckets in an incremental backup manner, thereby improving the backup efficiency of object storage buckets and solving the technical problem of low backup efficiency in the prior art when backing up object storage buckets.

[0076] In one optional embodiment, the snapshot processing system responds to a snapshot creation command and obtains a pre-snapshot, wherein the snapshot field of the pre-snapshot stores object metadata that has not yet been snapshotted. Then, the snapshot processing system persists the pre-snapshot to generate a new snapshot, wherein the new snapshot includes at least a snapshot number and a unique snapshot identifier corresponding to the snapshot number. The snapshot number of different snapshots is positively correlated with the generation time of each snapshot, and the unique snapshot identifier is a string whose byte order decreases as the snapshot number increases.

[0077] Optionally, during snapshot processing, each data write is redirected to the snapshot field reserved for the pre-snapshot, while old snapshot data remains in the snapshot field of the old snapshot. Therefore, upon receiving a snapshot creation instruction, the snapshot processing system only needs to persist the current pre-snapshot to generate a new snapshot. Simultaneously, the snapshot processing system also creates a new pre-snapshot, and subsequent data writes will be written to the snapshot field of the new pre-snapshot. The pre-snapshot can be understood as a temporary data set before the snapshot operation. Furthermore, when creating a new snapshot, the old snapshot changes from a read-write state to a read-only state.

[0078] In one alternative embodiment, Figure 7A schematic diagram of an optional snapshot creation process according to an embodiment of this application is shown. Figure 7 As shown, after receiving a snapshot creation request, the management platform persists the pre-snapshot data to the database and then notifies each object storage gateway to update the snapshot data (i.e., snapshot change). After updating the snapshot data, each object storage gateway writes objects using the new snapshot domain. Figure 7 The object storage gateways in the data are object storage gateway 1, object storage gateway 2, and object storage gateway 3.

[0079] In addition, each snapshot includes at least a snapshot number and a unique snapshot identifier corresponding to that snapshot number. The snapshot number can be understood as a snapshot ID, which is an incrementing integer. Since the latest data is usually obtained in actual production scenarios, the unique snapshot identifier should decrease in byte order as the snapshot ID increases, thus ensuring that the latest version of the object's metadata is listed first in the metadata database.

[0080] To ensure that the unique identifier of a snapshot decreases in byte order as the snapshot ID increases, this application employs a range inverse mapping method. The maximum value in the range of snapshot_id is subtracted from snapshot_id, and then a prefix is ​​added to the difference. The specific process is detailed in Table 1 below.

[0081] Table 1

[0082] snapshot_id prefix formula [1，16) "9” "9" + to_hex(15 - snapshot_id) [16，256) "8” "8" + to_hex(255 - snapshot_id) [256，65535) "7” "7"+to_hex(65535-snapshot_id) [65535，……] …… ……

[0083] The method to convert a decimal string to hexadecimal is to_hex(int).

[0084] In one optional embodiment, the snapshot processing system responds to a snapshot deletion command, determines the third and fourth snapshots corresponding to the object bucket, and determines a second object difference set based on the target log, wherein the third snapshot is the snapshot to be deleted, the fourth snapshot is the snapshot following the third snapshot, and the objects in the second object difference set are first objects, which are used to characterize objects that have been modified in the fourth snapshot compared to the third snapshot.

[0085] Optionally, after determining the second object difference set based on the target log, the snapshot processing system determines the historical object metadata and historical user data corresponding to the first object, and deletes the historical object metadata, historical user data, and the third snapshot.

[0086] Specifically, because snapshots have an inheritance relationship, when a snapshot is deleted, only objects modified by the later snapshot can be cleaned up. Figure 6For example, the third snapshot to be deleted could be snapshot 1, and the fourth snapshot could be snapshot 2. To determine the modified objects in the fourth snapshot compared to the third snapshot, the formula for the second object difference set can be set to delete(n), where n is the snapshot ID. Figure 6 For example, delete(snap1) = Tdiff(T1, T2) ∩ Tdiff(0, T1) = {delete{obj1}}, obj1 can be safely deleted.

[0087] It should be noted that the third and fourth snapshots can be any two adjacent snapshots from the multiple historical snapshots corresponding to the object bucket.

[0088] In one optional embodiment, the snapshot processing system responds to a snapshot rollback command by determining all historical snapshots at the current time. The snapshot rollback command includes at least a target snapshot number, and the historical snapshot corresponding to the target snapshot number is the target historical snapshot to which the system will roll back. The snapshot processing system then deletes the data in the pre-snapshot and rolls the current snapshot back to the target historical snapshot.

[0089] Optionally, after rolling back the current snapshot to the target historical snapshot, the snapshot processing system will also update the snapshot search path, which is used to characterize the association between multiple historical snapshots corresponding to the object bucket.

[0090] Furthermore, the snapshot search path is a data structure generated to quickly locate objects. It exists only in memory and is not persisted to disk. The snapshot search path is recalculated every time the bucket configuration is updated. Due to the dependencies between snapshots, from the user's perspective, the data in a particular snapshot is incomplete. That snapshot only contains the data from the end of the previous snapshot to the end of this snapshot. Only when all snapshots (a certain link) are combined does the final data state of the object storage bucket appear.

[0091] It should be noted that snapshot rollback can also be used in software testing. For example, after testing a new version of the software, testers can use snapshot rollback to roll the software back from the new version to the old version, thereby avoiding the problem of not being able to return to the old version state in time when the software encounters an anomaly in the new version, thus improving testing efficiency and testing stability.

[0092] Optional, Figure 8 A schematic diagram of a snapshot search path according to an embodiment of this application is shown, such as... Figure 8As shown, because snapshots can be rolled back, the dependencies between snapshots are disjointed relative to historical snapshots, necessitating the calculation of snapshot search paths. The dependencies between snapshots resemble a multi-way tree; since the path from the root node to any leaf node is always unique, the calculated snapshot search path is also unique. Furthermore, based on the snapshot search chain, objects not on the chain are filtered when retrieving object metadata to achieve the rolled-back state. For example, Figure 8 The image shows six snapshots: Snapshot 1, Snapshot 2, Snapshot 3, Snapshot 5, Snapshot 7, and Snapshot 8. These six snapshots form three chains: the first chain is: Snapshot 8, Snapshot 7, Snapshot 2, Snapshot 1; the second chain is: Snapshot 5, Snapshot 1; and the third chain is: Snapshot 3, Snapshot 2, Snapshot 1. Taking the third chain as an example, if Snapshot 5 is rolled back to Snapshot 1, the snapshot processing system will filter objects on the second chain and the first chain.

[0093] In one alternative embodiment, Figure 9 A schematic diagram of an optional snapshot rollback process according to an embodiment of this application is shown, such as... Figure 9 As shown, during snapshot rollback, the snapshot processing system persists snapshot data through the management platform and notifies the object storage gateway to make snapshot changes. Then, the object storage gateway deletes the data in the pre-snapshot and updates the snapshot data. This updating of snapshot data means rolling back the current snapshot to the target historical snapshot. The management platform automatically sends various snapshot processing commands to the object storage gateway. It's important to note that since snapshot rollback reverts from the latest snapshot to the target snapshot, and the current pre-snapshot is only a temporary state and does not belong to any specific snapshot, the data in the pre-snapshot needs to be deleted.

[0094] In one optional embodiment, in practical applications, there may be multiple object storage gateways. In a multi-object storage gateway scenario, due to the time difference between the snapshot processing instructions received by different object storage gateways from the management platform, the snapshot update times recorded by the multiple object storage gateways may differ. To avoid inaccurate snapshot comparisons caused by this difference, the snapshot processing system in this application determines the time when the first object storage gateway generates the first snapshot as the first time, and the time when the first object storage gateway generates the second snapshot as the second time. It also determines the time when the second object storage gateway generates the first snapshot as the third time and the time when the second object storage gateway generates the second snapshot as the fourth time. Then, the snapshot processing system detects the time order of the first time and the third time and determines the time with the earlier time order as the first target time. It also detects the time order of the second time and the fourth time and determines the time with the later time order as the second target time. Finally, the snapshot processing system determines the first object difference set between the first snapshot and the second snapshot based on the first target time and the second target time.

[0095] Optionally, the snapshot processing system determines a third set of difference objects based on the target log. The objects in the third set of difference objects are those that were modified between the first target time and the second target time. Then, the snapshot processing system determines the objects corresponding to the second snapshot as objects to be excluded. Finally, the snapshot processing system deletes the objects to be excluded from the third set of difference objects to obtain the first set of object difference objects.

[0096] Specifically, during the multi-object storage gateway snapshot comparison process, each time the object storage gateway updates snapshot data, it records two update times: the latest update time of the previous snapshot (latest) and the earliest update time of the latest pre-snapshot (earlist). The earlist cannot be overwritten, while the latest can. The formula for the set of differences between snapshots is diff(n, m), where n and m are snapshot IDs. The difference object between two points in time is Tdiff(T1, T2), where T1 and T2 are time points, where T1 is the first target time and T2 is the second target time. The set of values ​​not belonging to this interval is exclude(x), where x is the snapshot ID.

[0097] by Figure 10For example, assuming the first object storage gateway is object storage gateway 1, the second object storage gateway is object storage gateway 2, the first snapshot is snapshot 2, and the second snapshot is snapshot 3, then the first time is T2 earlist, the second time is the last generated T2 latest, the third time is T1 latest after the first update, and the fourth time is T3 earlist. By comparing the time order of the first time and the third time, we can determine that the first time T2 earlist is the first target time, and the second time is the second target time.

[0098] According to the timeline, the changes to objects and snapshots in the object bucket are as follows: Upload obj1, XGM1 changes snapshot 1 to snapshot 2, upload obj2, upload obj3, XGM2 changes snapshot 1 to snapshot 2, upload obj1, XGM2 changes snapshot 2 to snapshot 3, upload obj4, upload obj5, XGM1 changes snapshot 2 to snapshot 3, upload obj6. Using the formula diff(snap1, snap2) = Tdiff(T2 earlist, T2 latest) - exclude(snap2) = {put{obj2, obj5}, delete{obj1}}, snapshot deletion follows the same logic: delete(snap1) = T diff(snap1, snap2) ∩ Tdiff(0, snap1) = {delete{obj1}}, obj1 can be safely deleted.

[0099] As discussed above, traditional multi-version control manages data at the object level, generating a new version with each data modification, resulting in low space utilization. In batch backup scenarios, existing technologies typically perform full backups at the object storage bucket level, leading to low backup efficiency. This application, however, manages data at the bucket level with a time reference, enabling incremental backups based on data differences between snapshots, making batch backups simpler and more efficient. Furthermore, bucket snapshots allow users to create snapshots at any time and roll back to the state at the time of snapshot creation. In terms of processing time, snapshot creation, deletion, and rollback can all be performed within seconds. In terms of space, snapshots themselves occupy virtually no space, except for object data, supporting massive amounts of data and improving space utilization. In terms of specifications, managing snapshots at the bucket level with a time reference ensures compatibility with multiple versions.

[0100] Example 2

[0101] According to another aspect of the embodiments of this application, a snapshot processing apparatus based on an object storage bucket is also provided, wherein, Figure 11This is a schematic diagram of an optional snapshot processing apparatus according to an embodiment of this application. Figure 11 As shown, the device includes: a first determining module 1101, an acquiring module 1102, a second determining module 1103, and a backup module 1104.

[0102] The first determining module 1101 is used to respond to a snapshot comparison command through the object storage gateway to determine the first snapshot and the second snapshot corresponding to the object storage bucket, wherein the generation time of the first snapshot is earlier than the generation time of the second snapshot, and the snapshot comparison command is used to compare two snapshots with different generation times; the obtaining module 1102 is used to obtain the target log based on the generation time of the first snapshot and the generation time of the second snapshot, wherein the target log is stored in a preset area, and the target log is the object operation record sorted according to the completion time of the modification operation; the second determining module 1103 is used to determine the first object difference set between the first snapshot and the second snapshot based on the target log, wherein the first object difference set includes objects that differ between the second snapshot and the first snapshot, and the objects are the data stored in the object storage bucket; the backup module 1104 is used to perform incremental backup of the data generated by the object storage bucket in a target time period based on the first object difference set, wherein the target time period is the time interval between the generation time of the first snapshot and the generation time of the second snapshot.

[0103] It should be noted that the first determining module 1101, the obtaining module 1102, the second determining module 1103 and the backup module 1104 mentioned above correspond to steps S301 to S304 in the above embodiments. The four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0104] Optionally, the snapshot processing device further includes: a first acquisition module and a first generation module, wherein the first acquisition module is used to acquire a pre-snapshot in response to a snapshot creation instruction, wherein the snapshot field of the pre-snapshot stores object metadata that has not been snapshotted; the first generation module persists the pre-snapshot and generates a new snapshot, wherein the new snapshot includes at least a snapshot number and a snapshot unique identifier corresponding to the snapshot number, the snapshot number of different snapshots is positively correlated with the generation time of each snapshot, and the snapshot unique identifier is a string whose byte order decreases as the snapshot number increases.

[0105] Optionally, the snapshot processing apparatus further includes a third determining module and a fourth determining module. The third determining module is used to respond to a snapshot deletion command and determine a third snapshot and a fourth snapshot corresponding to the object storage bucket, wherein the third snapshot is the snapshot to be deleted, and the fourth snapshot is the snapshot following the third snapshot; the fourth determining module is used to determine a second object difference set based on the target log, wherein the objects in the second object difference set are first objects, and the first objects are used to characterize objects that have been modified in the fourth snapshot compared to the third snapshot.

[0106] Optionally, the snapshot processing device further includes a fifth determining module and a first deleting module. The fifth determining module is used to determine the historical object metadata and historical user data corresponding to the first object; the first deleting module is used to delete the historical object metadata, historical user data, and the third snapshot.

[0107] Optionally, the snapshot processing device further includes: a sixth determining module, a second deleting module, and a rollback module. The sixth determining module is used to respond to a snapshot rollback command and determine all historical snapshots at the current time, wherein the snapshot rollback command includes at least a target snapshot number, and the historical snapshot corresponding to the target snapshot number is the target historical snapshot to which the data is to be rolled back. The second deleting module is used to delete data in the pre-snapshot. The rollback module is used to roll back the current snapshot to the target historical snapshot.

[0108] Optionally, the snapshot processing apparatus further includes an update module for updating the snapshot search path, wherein the snapshot search path is used to characterize the association between multiple historical snapshots corresponding to an object storage bucket.

[0109] Optionally, the object storage gateway includes at least a first object storage gateway and a second object storage gateway. There is a time difference between the first and second object storage gateways when receiving instructions. The snapshot processing device further includes: a seventh determining module, an eighth determining module, a first detecting module, a second detecting module, and a ninth determining module. Specifically, the seventh determining module is used to determine the time when the first object storage gateway generates the first snapshot as a first time, and the time when the first object storage gateway generates the second snapshot as a second time; the eighth determining module is used to determine the time when the second object storage gateway generates the first snapshot as a third time, and the time when the second object storage gateway generates the second snapshot as a fourth time; the first detecting module is used to detect the time sequence between the first and third times, and determine the time that is earlier in the time sequence as a first target time; the second detecting module is used to detect the time sequence between the second and fourth times, and determine the time that is later in the time sequence as a second target time; and the ninth determining module is used to determine a first set of object differences between the first and second snapshots based on the first target time and the second target time.

[0110] Optionally, the aforementioned ninth determining module further includes: a first determining unit, a second determining unit, and a first deleting unit. The first determining unit is used to determine a third set of difference objects based on the target log, wherein the objects in the third set of difference objects are those whose modification operations occurred between the first target time and the second target time; the second determining unit is used to determine the objects corresponding to the second snapshot as objects to be excluded; and the first deleting unit is used to delete the objects to be excluded from the third set of difference objects, thus obtaining a first set of object difference objects.

[0111] Example 3

[0112] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program is configured to execute the snapshot processing method based on object bucket in Embodiment 1 above when it runs.

[0113] Example 4

[0114] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are configured to run the programs, wherein the programs are configured to execute the snapshot processing method based on object storage bucket in Embodiment 1 above.

[0115] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0116] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0120] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0121] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for snapshot processing based on an object storage bucket, characterized in that, The method comprises the following steps: In response to a snapshot comparison instruction, determine a first snapshot and a second snapshot corresponding to an object storage bucket through an object storage gateway, wherein the generation time of the first snapshot is earlier than the generation time of the second snapshot, and the snapshot comparison instruction is used to compare two snapshots with different generation times; According to the generation time of the first snapshot and the generation time of the second snapshot, obtain a target log, wherein the target log is stored in a preset area, and the target log is an object operation record sorted according to the completion time of a modification operation; According to the target log, determine a first object difference set between the first snapshot and the second snapshot, wherein the first object difference set includes objects that are different between the second snapshot and the first snapshot, and the objects are data stored in the object storage bucket; According to the first object difference set, perform incremental backup on data generated by the object storage bucket in a target time period, wherein the target time period is the time interval between the generation time of the first snapshot and the generation time of the second snapshot; The object storage gateway comprises at least a first object storage gateway and a second object storage gateway, and the first object storage gateway and the second object storage gateway have a time difference when receiving instructions, and the method comprises the following steps: Determine that the first object storage gateway generates the first snapshot at a first time, and the first object storage gateway generates the second snapshot at a second time; Determine that the second object storage gateway generates the first snapshot at a third time, and the second object storage gateway generates the second snapshot at a fourth time; Detect the time sequence of the first time and the third time, and determine that the time earlier in the time sequence is a first target time; Detect the time sequence of the second time and the fourth time, and determine that the time later in the time sequence is a second target time; According to the first target time and the second target time, determine a first object difference set between the first snapshot and the second snapshot.

2. The method of claim 1, wherein, The method further comprises the following steps: In response to a snapshot creation instruction, obtain a pre-snapshot, wherein the object metadata that has not undergone a snapshot operation is stored in the snapshot domain of the pre-snapshot; Persist the pre-snapshot to generate a new snapshot, wherein the new snapshot comprises at least a snapshot number and a snapshot unique identifier corresponding to the snapshot number, the snapshot numbers of different snapshots are positively correlated with the generation times of each snapshot, and the snapshot unique identifier is a byte sequence that decreases with the increase of the snapshot number.

3. The method of claim 1, wherein, The method further comprises the following steps: In response to a snapshot deletion instruction, determine a third snapshot and a fourth snapshot corresponding to the object storage bucket, wherein the third snapshot is a snapshot to be deleted, and the fourth snapshot is the next snapshot of the third snapshot; According to the target log, determine a second object difference set, wherein the objects in the second object difference set are first objects, and the first objects are used to represent objects that have been modified in the fourth snapshot compared with the third snapshot.

4. The method of claim 3, wherein, After determining the second object difference set according to the target log, the method further comprises: determining historical object metadata and historical user data corresponding to the first object; deleting the historical object metadata, the historical user data and the third snapshot.

5. The method of claim 2, wherein, The method further comprises: in response to a snapshot rollback instruction, determining all historical snapshots at a current time, wherein the snapshot rollback instruction at least includes a target snapshot number, and the historical snapshot corresponding to the target snapshot number is a target historical snapshot to be rolled back to; deleting data in the pre-snapshot; rolling back the current snapshot to the target historical snapshot.

6. The method of claim 5, wherein, After rolling back the current snapshot to the target historical snapshot, the method further comprises: updating a snapshot search path, wherein the snapshot search path is used to represent the association relationship between a plurality of historical snapshots corresponding to the object storage bucket.

7. The method of claim 1, wherein, Determining the first object difference set between the first snapshot and the second snapshot according to the first target time and the second target time comprises: determining a third difference object set according to the target log, wherein the objects in the third difference object set are objects that have been subjected to modification operations between the first target time and the second target time; determining that the corresponding object in the second snapshot is a to-be-excluded object; deleting the to-be-excluded object from the third difference object set to obtain the first object difference set.

8. An object storage bucket based snapshot processing apparatus, comprising: The device comprises: a first determining module configured to determine, by an object storage gateway in response to a snapshot comparison instruction, a first snapshot and a second snapshot corresponding to an object storage bucket, wherein the generation time of the first snapshot is earlier than the generation time of the second snapshot, and the snapshot comparison instruction is used to compare two snapshots with different generation times; an obtaining module configured to obtain a target log according to the generation time of the first snapshot and the generation time of the second snapshot, wherein the target log is stored in a pre-set area, and the target log is an object operation record sorted according to the completion time of a modification operation; a second determining module configured to determine a first object difference set between the first snapshot and the second snapshot according to the target log, wherein the first object difference set includes objects that have differences between the second snapshot and the first snapshot, and the objects are data stored in the object storage bucket; a backup module configured to perform incremental backup on data generated by the object storage bucket in a target time period according to the first object difference set, wherein the target time period is a time interval between the generation time of the first snapshot and the generation time of the second snapshot; The object storage gateway includes at least a first object storage gateway and a second object storage gateway, the first object storage gateway and the second object storage gateway have a time difference when receiving an instruction, the device is further configured to determine that a time when the first object storage gateway generates the first snapshot is a first time, and a time when the first object storage gateway generates the second snapshot is a second time; determine that a time when the second object storage gateway generates the first snapshot is a third time, and a time when the second object storage gateway generates the second snapshot is a fourth time; detect a time sequence of the first time and the third time, and determine that a time in front of the time sequence as a first target time; detect a time sequence of the second time and the fourth time, and determine that a time in back of the time sequence as a second target time; and determine a first object difference set between the first snapshot and the second snapshot according to the first target time and the second target time.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is configured to execute the object storage bucket based snapshot processing method in any one of claims 1 to 7 when running.

10. An electronic device, comprising: The electronic device includes one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement a program for running, and the program is configured to execute the object storage bucket based snapshot processing method in any one of claims 1 to 7 when running.

Images