Data processing method, device, equipment, program product and storage medium

Abstract

The present application discloses a data processing method, apparatus, device, program product and storage medium, the method comprising: obtaining first identification group information of first snapshot data generated after modifying a first file and metadata corresponding to the first file; determining second identification group information of second snapshot data stored in the metadata; when third identification group information other than the first identification group information is found from the second identification group information, searching for third snapshot data corresponding to the third identification group information from the second snapshot data, and deleting the third snapshot data; data can be quickly recovered, thereby reducing the overhead on file system performance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment, program product and storage medium. Background Art

[0002] Snapshots can save all the data in the file system at the moment, and are often used in scenarios such as data backup, data recovery, fault migration, and data analysis. However, when deleting a snapshot, it is necessary to scan all the data in the entire file system, which is time-consuming, slow to reclaim space, and has high system overhead. Summary of the invention

[0003] To solve the existing technical problems, the embodiments of the present application provide a data processing method, device, equipment, program product and storage medium, which can quickly delete data according to the identification group information of the data, and take a short time, thereby reducing the overhead of the file system.

[0004] To achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a data processing method, characterized in that the method includes:

[0006] Acquire first identification group information of first snapshot data generated after modifying the first file and metadata corresponding to the first file;

[0007] Determine second identification group information of second snapshot data stored in the metadata;

[0008] When third identification group information other than the first identification group information is found in the second identification group information, third snapshot data corresponding to the third identification group information is searched in the second snapshot data, and the third snapshot data is deleted.

[0009] In the above solution, deleting the third snapshot data includes:

[0010] Acquire fourth snapshot data; the fourth snapshot data is a sub-snapshot of the third snapshot data;

[0011] Determine whether the fourth snapshot data has a sub-snapshot dependency;

[0012] If the fourth snapshot data does not have the sub-snapshot dependency, deleting the third snapshot data;

[0013] In the case that the fourth snapshot data has the sub-snapshot dependency, the first portion of snapshot data that does not have the sub-snapshot dependency is searched from the third snapshot data, and the first portion of snapshot data is deleted.

[0014] In the above solution, determining whether the fourth snapshot data has a sub-snapshot dependency includes:

[0015] Acquire at least one tag information of the fourth snapshot data;

[0016] In a case where each of the marking information in the at least one marking information is the same as the preset information, determining that the fourth snapshot data has no sub-snapshot dependency;

[0017] When the first marking information in the at least one marking information is different from the preset information, it is determined that the fourth snapshot data has the sub-snapshot dependency.

[0018] In the above scheme, the method further includes:

[0019] In the case where it is determined that the fourth snapshot data has the sub-snapshot dependency, marking the third snapshot data;

[0020] When an update operation on the first file is detected, the first marking information is modified to the preset information, the second part of the snapshot data corresponding to the first marking information in the third snapshot data is deleted, and the marking of the third snapshot data is removed.

[0021] In the above scheme, the method further includes:

[0022] When fourth identification group information other than the second identification group information is found in the first identification group information, fifth snapshot data corresponding to the fourth identification group information is found in the first snapshot data, and the fifth snapshot data is created.

[0023] In the above solution, the creating the fifth snapshot data includes:

[0024] Filling at least one tag information of the fourth snapshot data into a blank tag table corresponding to the fifth snapshot data to obtain second tag information corresponding to the fifth snapshot data;

[0025] The at least one mark information is cleared, and the second mark information is stored in metadata corresponding to the fifth snapshot data to create the fifth snapshot data.

[0026] The present application provides a data processing device, the device comprising:

[0027] An acquiring unit, configured to acquire first identification group information of first snapshot data generated after modifying the first file and metadata corresponding to the first file;

[0028] a determining unit, configured to determine second identification group information of second snapshot data stored in the metadata;

[0029] The deleting unit is used to search the second snapshot data for third snapshot data corresponding to the third identification group information when the third identification group information other than the first identification group information is found in the second identification group information, and delete the third snapshot data.

[0030] An embodiment of the present application provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0031] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0032] An embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0033] The embodiment of the present application provides a data processing method, device, equipment, program product and storage medium. The method includes: obtaining first identification group information of first snapshot data generated after modifying a first file and metadata corresponding to the first file; determining second identification group information of second snapshot data stored in the metadata; in the case where third identification group information other than the first identification group information is found from the second identification group information, searching for third snapshot data corresponding to the third identification group information from the second snapshot data, and deleting the third snapshot data; adopting the scheme of the embodiment of the present application, by using the first identification group information to represent the first snapshot data generated after modifying the first file, using the second identification group information to represent the second snapshot data stored in the metadata corresponding to the first file, and finding the third identification group information other than the first identification group information in the second identification group information, the third snapshot data to be deleted can be directly found in the second snapshot data through the third identification group information, so that after the snapshot data is represented by the identification group information, the snapshot data to be deleted can be directly found through the identification group information without scanning all the snapshot data for searching, which saves time and recovers data quickly, thereby reducing the overhead of the file system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of a process of creating and deleting a snapshot provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a process for creating a snapshot provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of a snapshot directory tree provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of a snapshot relationship provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of a snapshot relationship after a first write operation is provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of a snapshot relationship after a second write operation is provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of a snapshot relationship after a partial deletion operation provided in an embodiment of the present application;

[0042] Fig. 9 A schematic diagram of a snapshot relationship after a deletion operation provided in an embodiment of the present application;

[0043] Fig.10 A schematic diagram of a snapshot relationship after a rollback operation provided in an embodiment of the present application;

[0044] Fig.11 A schematic diagram of a process of scanning and deleting snapshots provided in an embodiment of the present application;

[0045] Fig.12 A schematic diagram of a data processing device provided in an embodiment of the present application;

[0046] Fig.13 A schematic diagram of a hardware entity structure of a data processing device in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] A snapshot is a consistency point of a distributed file system at a certain moment, which saves all the data in the file system at that moment. It is often used in daily data backup, data recovery, fault migration, and data analysis scenarios. However, after creating a snapshot, modifying the existing data before creating the snapshot will generate a large amount of snapshot data backup. When deleting a snapshot, the traditional approach is always to scan all the data in the entire file system. However, there is always a huge amount of data in a distributed file system. Deleting snapshots by scanning has the disadvantages of long scanning time, slow space recovery, and high system overhead.

[0049] Based on this, an embodiment of the present application provides a data processing method, apparatus, device, program product and storage medium. When a file is modified, the snapshot data is quickly recovered by comparing the identification group information of the snapshot data generated after the file is modified with the identification group information of the snapshot data stored in the metadata.

[0050] The present application embodiment provides a data processing method. Figure 1 A flowchart of a data processing method provided in an embodiment of the present application; Figure 1 As shown, the method includes:

[0051] Step 101: Acquire first identification group information of first snapshot data generated after modifying a first file and metadata corresponding to the first file.

[0052] Step 102: Determine second identification group information of second snapshot data stored in metadata.

[0053] Step 103: When third identification group information other than the first identification group information is found in the second identification group information, third snapshot data corresponding to the third identification group information is searched in the second snapshot data, and the third snapshot data is deleted.

[0054] It should be noted that, in practical applications, the data processing method can be understood as a snapshot data processing method of the file system.

[0055] In step 101, the first file can be any file in the file system, which can be determined according to actual conditions and is not limited here. Modification of the first file can be understood as replacing, deleting, and / or adding data of the first file, and the specific modification process can be determined according to actual conditions and is not limited here. The first identification group information of the first snapshot data can be understood as a snapshot identity (Identity, ID) set (snapid set) of the first snapshot data.

[0056] Acquiring metadata corresponding to the first file can be understood as acquiring identification information corresponding to the first file when modifying the first file, passing the identification information corresponding to the first file to the object data management process (Total Data Solutions, TDS) data management part, and the TDS data management part acquiring metadata corresponding to the first file based on the identification information. It should be noted that the identification information corresponding to the first file can be recorded as oid in actual applications.

[0057] In step 102, the second identification group information can be understood as a snapshot ID set of the second snapshot data stored in the metadata. For ease of understanding, Table 1 is the content stored in the metadata, as shown in Table 1:

[0058] Table 1

[0059]

[0060] It should be noted that metadata is stored in the form of key-value pairs (KV). The left column of Table 1 represents Key, and the right column of Table 1 represents Value.

[0061] Key represents the identification information of the metadata stored in the data management part, including metadata type (Type), version (Version), data storage pool (Poolid) and data storage object identifier (Oid); the specific explanation is as follows:

[0062] Metadata type: indicates that it is a file system data (base) object element kv;

[0063] Version: indicates the version of kv for subsequent upgrades;

[0064] Data storage pool: represents different data storage pools;

[0065] Data storage object ID: represents different data storage objects.

[0066] Value represents the data content related to the Key, including the snapshot ID set (Snapid set), the number of snapshot IDs (Snapid num), the data mark of the snapshot header data (HEAD bitmap), and the snapshot data size (Size); the specific explanations are as follows:

[0067] Snapshot ID set: represents the snapshot combination of the object;

[0068] Snapshot ID number: indicates the number of snapshots that exist for the object;

[0069] Data tag of snapshot header data: indicates the data distribution description of the base file system;

[0070] Snapshot data size: indicates the size of the data object.

[0071] In step 103, when the third identification group information other than the first identification group information is found from the second identification group information, the third snapshot data corresponding to the third identification group information is searched from the second snapshot data, and the third snapshot data is deleted; in practical applications, it can be understood that, when the first identification group information is {1,2,3} and the second identification group information is {1,2,3,4}, the third identification group information can be understood as snap4, and the snapshot data of snap4 needs to be deleted.

[0072] By adopting the solution of the embodiment of the present application, by adding identification group information to the snapshot data, searching for the identification group information corresponding to the snapshot data stored in the file metadata according to the identification group information corresponding to the snapshot data after the file is modified, the snapshot data can be quickly recovered, thereby reducing the overhead of file system performance.

[0073] In one embodiment, deleting the third snapshot data includes: obtaining the fourth snapshot data; the fourth snapshot data is a sub-snapshot of the third snapshot data; determining whether the fourth snapshot data has a sub-snapshot dependency; if the fourth snapshot data has no sub-snapshot dependency, deleting the third snapshot data; if the fourth snapshot data has a sub-snapshot dependency, searching the third snapshot data for the first portion of snapshot data that has no sub-snapshot dependency, and deleting the first portion of snapshot data.

[0074] The fourth snapshot data may be understood as snapshot header data. In practical applications, the snapshot header data may be recorded as HEAD data.

[0075] Determining whether the fourth snapshot data has a sub-snapshot dependency can be understood as determining whether the fourth snapshot data has a sub-snapshot dependency through at least one tag information of the fourth snapshot data; specifically: when each tag information in at least one tag information is the same as preset information, it is determined that the fourth snapshot data has no sub-snapshot dependency; when the first tag information in at least one tag information is different from the preset information, it is determined that the fourth snapshot data has the sub-snapshot dependency.

[0076] When the fourth snapshot data has no sub-snapshot dependency, deleting the third snapshot data can be understood as deleting all snapshot data of the third snapshot data when the fourth snapshot data has no sub-snapshot dependency.

[0077] In the case where the fourth snapshot data has a sub-snapshot dependency, the first part of the snapshot data that has no sub-snapshot dependency is searched from the third snapshot data and the first part of the snapshot data is deleted. It can be understood that, in the case where the fourth snapshot data has a sub-snapshot dependency, the first part of the snapshot data that has no sub-snapshot dependency in the third snapshot data is deleted, and the snapshot data with sub-snapshot dependencies is retained.

[0078] In one embodiment, determining whether fourth snapshot data has a sub-snapshot dependency includes: obtaining at least one tag information of the fourth snapshot data; determining that the fourth snapshot data has no sub-snapshot dependency when each tag information in the at least one tag information is the same as preset information; and determining that the fourth snapshot data has a sub-snapshot dependency when the first tag information in the at least one tag information is different from the preset information.

[0079] The at least one mark information may be understood as data mark information of the fourth snapshot data, and the data mark information may be recorded as a bitmap in practical applications.

[0080] In the file system, the bitmap of the file is saved in the metadata of the data to record the valid data bits. When creating a snapshot, you only need to record the bitmap indicating the valid data mark in the metadata of the snapshot to indicate which valid data intervals are on the snapshot. When storing data, the data block is marked by recording one bit every 8k. For ease of understanding, Table 2 is an example table of the bitmap, as shown in Table 2:

[0081] Table 2

[0082]

[0083] In Table 2, from left to right, they represent the data of [0-8K], [8K-16K], [16K-24K], [24K-32K], [32K-40K], [40K-48K], [48K-56K], [56K-64K], [64K-72K] and [72K-80K], among which, if data is written in [0-8K], [8K-16K], [24K-32K], [32K-40K] and [56K-64K], it is recorded as 1; if data is not written in the other positions, it is recorded as 0.

[0084] The at least one mark information can also be understood as the value written in each position in Table 2, that is, bit0 and bit1 are 1, bit2 is 0, bit3 and bit4 are 1, bit5 and bit6 are 0, bit7 is 1, and bit8 and bit9 are 0.

[0085] It should be noted that the content and quantity actually written into the bitmap are determined according to the file data in the file system and are not limited here.

[0086] The preset information may be determined according to actual conditions and is not limited here. As an example, the preset information is 1. The first marking information may be understood as a bit whose value is not 1 in the bitmap of the fourth snapshot data.

[0087] When each tag information in at least one tag information is the same as the preset information, it is determined that the fourth snapshot data has no sub-snapshot dependency; when the first tag information in at least one tag information is different from the preset information, it is determined that the fourth snapshot data has sub-snapshot dependency; it can be understood that, when all the values ​​in the bitmap in the fourth snapshot data are 1, it is determined that the fourth snapshot data has no sub-snapshot dependency; when there is a value in the bitmap in the fourth snapshot data that is not 1, it is determined that the fourth snapshot data has sub-snapshot dependency.

[0088] In one embodiment, the method also includes: when it is determined that the fourth snapshot data has a sub-snapshot dependency, marking the third snapshot data; when an update operation on the first file is detected, modifying the first marking information to preset information, deleting the second part of the snapshot data corresponding to the first marking information in the third snapshot data, and unmarking the third snapshot data.

[0089] Wherein, when it is determined that the fourth snapshot data has a sub-snapshot dependency, the third snapshot data is marked, which can be understood as marking the third snapshot data when the fourth snapshot data has a sub-snapshot dependency, that is, the third snapshot data needs to be deleted. The method of marking the third snapshot data can be determined according to actual conditions and is not limited here. As an example, the third snapshot data can be marked by color.

[0090] The update operation of the first file can be determined according to actual conditions and is not limited here. As an example, the update operation of the first file can be to modify the first file again to overwrite the previous content, or to partially delete the first file, or to completely delete the first file, or to restore the modified first file to the first file before modification, etc.

[0091] The first marking information is modified to the preset information, the second part of the snapshot data corresponding to the first marking information in the third snapshot data is deleted, and the marking of the third snapshot data is released; it can be understood that the bit position in the bitmap of the fourth snapshot data that is not 1 is modified to 1. After being modified to 1, the sub-snapshot dependency of the fourth snapshot data can be released, and the second part of the snapshot data with sub-snapshot dependency in the third snapshot data is deleted, thereby releasing the dependency of the third snapshot data.

[0092] It should be noted that, when the fourth snapshot data has a sub-snapshot dependency and the first file is not updated, the third snapshot data needs to be scanned to delete the third snapshot data.

[0093] If the sixth snapshot data exists, and the sixth snapshot data is a sub-snapshot of the third snapshot data, the fourth snapshot data is a sub-snapshot of the sixth snapshot data, and both the sixth snapshot data and the fourth snapshot data have sub-snapshot dependencies, but the third snapshot data needs to be deleted, it is still necessary to first release the sub-snapshot dependency of the fourth snapshot data, and then delete the sixth snapshot data to release the sub-snapshot dependency of the sixth snapshot data, and finally delete the third snapshot data.

[0094] In one embodiment, the method further includes: when fourth identification group information other than the second identification group information is found in the first identification group information, searching the first snapshot data for fifth snapshot data corresponding to the fourth identification group information, and creating the fifth snapshot data.

[0095] The fourth identification group information can be understood as a snapshot ID. When the fourth identification group information other than the second identification group information is found from the first identification group information, the fifth snapshot data corresponding to the fourth identification group information is found from the first snapshot data, and the fifth snapshot data is created. For example, when the first identification group information is {1,2,3,4,5} and the second identification group information is {1,2,3,4}, the fourth identification group information is snap5, and the snapshot data of snap5 needs to be created.

[0096] In one embodiment, creating fifth snapshot data includes: filling at least one tag information of the fourth snapshot data into a blank tag table corresponding to the fifth snapshot data to obtain second tag information corresponding to the fifth snapshot data; clearing at least one tag information and storing the second tag information in metadata corresponding to the fifth snapshot data to create the fifth snapshot data.

[0097] The blank mark table corresponding to the fifth snapshot data can be understood as a blank bitmap table of the fifth snapshot data. Filling at least one mark information of the fourth snapshot data into the blank mark table corresponding to the fifth snapshot data to obtain the second mark information corresponding to the fifth snapshot data can be understood as copying the information in the bitmap table of the fourth snapshot data to the blank bitmap table of the fifth snapshot data.

[0098] At least one mark information is cleared, and the second mark information is stored in the metadata corresponding to the fifth snapshot data to create the fifth snapshot data; it can be understood that the information in the bitmap table of the fourth snapshot data is cleared, and the information in the bitmap table of the fifth snapshot data is saved in the metadata of the fifth snapshot data.

[0099] For ease of understanding, Table 3 shows the contents stored in the metadata of the fifth snapshot data, as shown in Table 3:

[0100] Table 3

[0101]

[0102] It should be noted that snapshot metadata is also stored in the form of KV. The left column of Table 3 represents Key, and the right column of Table 3 represents Value.

[0103] Key represents the identification information of snapshot metadata stored in the data management part, including snapshot data type (Type), version (Version), data storage pool (Poolid) and data storage object identifier (Oid); the specific explanation is as follows:

[0104] Snapshot data type: represents the kv of the snapshot object element;

[0105] Version: indicates the version of kv for subsequent upgrades;

[0106] Data storage pool: represents different data storage pools;

[0107] Data storage object ID: represents different data storage objects.

[0108] Value represents the data content related to the Key, including the snapshot ID (Snapid), the data mark in the snapshot data (bitmap), and the data mark deleted in the snapshot data (Delete_bitmap); the specific explanation is as follows:

[0109] Snapshot ID: metadata indicating which snapshot ID it is;

[0110] Data tags in snapshot data: indicate the distribution of valid data in the snapshot;

[0111] Deleted data mark in snapshot data: indicates the deleted snapshot data, used to mark the deleted snapshot.

[0112] It should be noted that after at least one mark information of the fourth snapshot data is cleared, if there is new data written, it will continue to be recorded in the bitmap of the fourth snapshot data.

[0113] For ease of understanding, in actual application, the data processing method is exemplified as a method for accelerating deletion of snapshot data by a file system, which is specifically described as follows:

[0114] The metadata management process (Taobao File System, TFS) can obtain all existing snapshots on the file when modifying data, and pass the snapshot combination to the TDS data management part. The TDS data management part determines whether the data object needs to create or delete a snapshot based on its own metadata information. Figure 2 A schematic diagram of a process of creating and deleting a snapshot provided in an embodiment of the present application; Figure 2 As shown; the following will be described in detail:

[0115] 1. The metadata management process creates and deletes snapshot operations.

[0116] It should be noted that, in the execution of a specific process, when creating and deleting snapshot operations, the metadata management process passes the object identifier and the snapshot identifier set to the data management part in the object data management process.

[0117] 2. The data management part in the object data management process receives the object identifier and snapshot identifier set sent by the metadata management process when creating and deleting snapshot operations.

[0118] 3. Get the metadata information of the object based on the object identifier.

[0119] It should be noted that the data management part in the object data management process obtains the metadata information of the object according to the object identifier.

[0120] 4. Get the saved snapshot information from the metadata information.

[0121] 5. Determine whether to create or delete a snapshot based on the snapshot identifier set input by the business.

[0122] 6. If the snapshot information in the object metadata information contains the snapshot identifier but the snapshot identifier set does not exist, it means that the snapshot is to be deleted.

[0123] 7. If the snapshot information in the object metadata does not contain the snapshot identifier but the snapshot identifier set exists, it means that a snapshot is to be created.

[0124] 8. Create or delete the snapshot and return it to the client.

[0125] 9. The metadata management process creates and deletes snapshot operations.

[0126] Snapshot creation process: When the snapshot delivered by the business is not included in the snapshot combination that has been created for the object, it means that the snapshot needs to be created. Figure 3 A schematic diagram of a process of creating a snapshot provided in an embodiment of the present application; Figure 3 As shown, Figure 3 The specific process is as follows:

[0127] 1. Calculate the snapshot 1 required based on the snapshot identification parameters delivered by the business.

[0128] 2. Copy the data mark message recorded in the snapshot header of the file system.

[0129] 3. Clear the data mark message in the snapshot header to 0.

[0130] 4. Save the data tag information in the copied snapshot header to the metadata of snapshot 1.

[0131] 5. Insert snapshot 1 into the snapshot tree structure.

[0132] The above process can also be explained as: copy the bitmap information recorded in the current HEAD to snapshot 1, because when snapshot 1 does not exist, all valid data bits are saved in the bitmap of HEAD; clear the bitmap information recorded in HEAD, and the newly written data will continue to be marked in HEAD to prepare for the next snapshot; save the bitmap to the metadata corresponding to snapshot 1; add the snapshot information to the tree structure maintained by HEAD to form a snapshot directory tree.

[0133] Figure 4 A schematic diagram of a snapshot directory tree provided in an embodiment of the present application; Figure 4 As shown, snapshot 2 is a sub-snapshot of snapshot 1, snapshot 3 is a sub-snapshot of snapshot 2, snapshot 4 is a sub-snapshot of snapshot 3, snapshot 5 is a sub-snapshot of snapshot 4, and the snapshot header is a sub-snapshot of snapshot 5. It should be noted that snapshot 1 can be recorded as S1 in actual applications, snapshot 2 can be recorded as S2 in actual applications, snapshot 3 can be recorded as S3 in actual applications, snapshot 4 can be recorded as S4 in actual applications, snapshot 5 can be recorded as S5 in actual applications, and the snapshot header can be recorded as HEAD in actual applications.

[0134] Delete snapshot: When the snapshot combination passed when the business modifies the file for the first time is {1,2,3,4}, and the snapshot passed in the second modification action is {2,3,4}, it means snapshot 1, that is, snapshot 1, is to be deleted. The process of deleting snapshot data is as follows: Get the snapshot bitmap corresponding to the snapshot, and determine whether its bitmap has sub-snapshot dependencies; delete the data that has no sub-snapshot dependencies, and delete the snapshot if the snapshot data can be completely deleted; if there is still some data dependency, temporarily retain the data, mark the snapshot for deletion, and continue to delete the undeleted data when the dependency is removed.

[0135] Figure 5 A schematic diagram of a snapshot relationship provided in an embodiment of the present application; Figure 5As shown in the figure, snapshot 1 is the parent snapshot of snapshot 2, and snapshot 2 is the parent snapshot of the snapshot header; that is, snapshot 2 is the child snapshot of snapshot 1, and the snapshot header is the child snapshot of snapshot 2. In the snapshot, only one copy of the same data is retained. Therefore, when reading the data from 0 to 8k of the snapshot header, since the bitmap of the snapshot header is 0, the snapshot dependency will be searched when reading the data. At this time, the data read will read the data of snapshot 2 on which it depends. The values ​​of bit0-bit1 of the snapshot header data are 0, and the dependency of bit0-bit1 is snapshot 2. The values ​​of bit2-bit3 are 1, and bit2-bit3 does not depend on snapshot 2. The values ​​of bit4-bit6 are 0, and bit4-bit6 depends on snapshot 2. For snapshot 2, since all its bits are set to 1, it has no dependency on snapshot 1. Therefore, when deleting snapshot 1, since snapshot 2 has no dependency on the data of snapshot 1, its data can be completely cleared when deleting snapshot 1, and the metadata information of the snapshot can be deleted. However, when deleting snapshot 2, since the bits in the snapshot header are dependent on snapshot 2, only the overlapping and non-dependent data bits are deleted when deleting snapshot 2, and then snapshot 2 is marked for deletion. When the snapshot header and its dependent bits are filled, the rest are deleted. In practical applications, snapshot 1 can be recorded as snap1, snapshot 2 can be recorded as snap2, and the snapshot header can be recorded as HEAD.

[0136] Remove the mark-for-deletion dependency through a write operation: For a snapshot that is marked for deletion, some snapshot data is temporarily not recovered due to the dependency of the child snapshot or base file system. Therefore, when the dependent data area is filled with new data, this dependency will be removed and the data in this snapshot space can be deleted. Figure 6 A schematic diagram of a snapshot relationship after a first write operation is provided in an embodiment of the present application; Figure 6 As shown in the figure, when data is written to bit4-bit6 in the snapshot header, that is, the value of bit4-bit6 is overwritten to 1, its bit will be set, and the data space of bit4-bit6 in snapshot 2 will be reclaimed. Figure 7 A schematic diagram of a snapshot relationship after a second write operation is provided in an embodiment of the present application; Figure 7 As shown in the figure, when bit0-bit1 in the snapshot header is also written with valid data, that is, the value of bit0-bit1 is also overwritten with 1, the snapshot header is completely freed from the dependency on snapshot 2, and the space of bit0-bit1 in snapshot 2 can be reclaimed. At the same time, the snapshot metadata represented by snapshot 2 will also be reclaimed. At this point, the marked deletion is finally converted to complete deletion.

[0137] The Truncate operation removes the mark for deletion dependency: Figure 8A schematic diagram of a snapshot relationship after a partial deletion operation is provided in an embodiment of the present application; Figure 8 As shown in the figure, when the snapshot header is truncated to 0 or smaller than the space of snapshot 2, the dependency on some data of snapshot 2 will be removed. When it is completely truncated to 0, the dependency on snapshot 2 will be completely removed, and all bits in the snapshot header will be overwritten to 0. At this time, the data in the space of snapshot 2 except the space of bit2-bit3 can be recycled and the snapshot metadata information represented by snapshot 2 can be deleted.

[0138] The Delete operation removes the mark for deletion dependency: Fig. 9 A schematic diagram of a snapshot relationship after a deletion operation is provided in an embodiment of the present application; Fig. 9 As shown in the figure, when the business completely deletes the data objects of the base file system, all the bits in the snapshot header are overwritten to 0. At this time, the snapshot header is no longer dependent on snapshot 2. The data space other than bit2-bit3 in snapshot 2 can be reclaimed and the snapshot metadata corresponding to snapshot 2 will also be deleted.

[0139] The rollback operation removes the mark for deletion of dependencies: Fig.10 A schematic diagram of a snapshot relationship after a rollback operation is provided in an embodiment of the present application; Fig.10 As shown in the figure, when the user or business rolls back the base file system to snapshot 1, snapshot 2, which is marked for deletion, will have no child snapshot data dependency. At this time, its data can be completely recovered and the snapshot metadata information corresponding to snapshot 2 can be deleted. Snapshot 1 is the parent snapshot of snapshot 2, and snapshot 1 is also the parent snapshot of the snapshot header.

[0140] Scan and delete snapshots: When the business is no longer accessing the data before the snapshot is deleted, it is still necessary to rely on snapshot data scanning to complete the recovery of the deleted snapshot data space. Fig.11 A schematic diagram of a process of scanning and deleting snapshots provided in an embodiment of the present application; Fig.11 As shown, Fig.11 The detailed description is as follows:

[0141] 1. The metadata management process deletes the snapshot operation.

[0142] 2. The object data management process receives the snapshot identifier of the snapshot to be deleted sent by the metadata management process when deleting the snapshot operation.

[0143] 3. The object data management process receives the parameters and starts an asynchronous task to scan and delete snapshots.

[0144] 4. The metadata data process deletes the snapshot metadata and ends.

[0145] 5. The metadata management process deletes the snapshot and ends.

[0146] 6. The object data management process determines whether the object contains the snapshot identifier of the snapshot to be deleted.

[0147] It should be noted that when the object contains the snapshot identifier of the snapshot to be deleted, go to step 7; when the object does not contain the snapshot identifier of the snapshot to be deleted, go to step 8.

[0148] 7. Delete snapshot data (completely delete or mark for deletion).

[0149] It should be noted that after deleting snapshot data of a certain object, it is necessary to go to 8 to continue scanning the next object.

[0150] 8. Continue scanning the next object.

[0151] 9. Scanning completed.

[0152] It should be noted that, in the specific process, it is necessary to determine whether the next object has been scanned. If the next object has not been scanned, proceed to step 6 to determine whether the object contains the snapshot identifier. If the next object has been scanned, proceed to step 10.

[0153] 10. Delete the snapshot ID of the snapshot to be deleted.

[0154] The above process can also be explained as follows: when TFS performs a snapshot deletion operation, the SnapDid to be deleted is passed to TDS. TDS receives the SnapDid to be deleted and starts an asynchronous task, that is, it starts scanning and deleting snapshots to determine whether the object contains the SnapDid. If the object contains the SnapDid, the corresponding snapshot data is deleted. During the deletion process, if the sub-snapshot data of the snapshot data has no sub-snapshot dependency, it is completely deleted; if the sub-snapshot data of the snapshot data has a sub-snapshot dependency, it is marked for deletion; if the SnapDid is not contained, the next object is scanned until an object containing the SnapDid is scanned and the snapshot data is deleted. Among them, when TDS starts scanning and deleting snapshots, it sends a feedback message to TFS to inform TFS that it has scanned the snapshot for deletion. After TFS receives the feedback message, the deletion of the snapshot metadata ends.

[0155] According to the method of the embodiment of the present application, when a user modifies or deletes a file, it is only necessary to bring in a valid snapid queue, and the distributed storage data portion can complete the rapid recovery of the snapshot data according to the snapid queue, and the expensive storage space can be quickly recovered without waiting for the completion of the snapshot scanning task; the recovery of the distributed storage space can be completed simultaneously in the input and output (io) path of the user's business without affecting the normal business io performance, thereby achieving the advantages of fast storage space recovery, no user business perception, and good performance.

[0156] The present application also provides a data processing device. Fig.12 A schematic diagram of a data processing device provided in an embodiment of the present application; Fig.12 As shown, the data processing device 1200 includes:

[0157] An acquiring unit 1201 is configured to acquire first identification group information of first snapshot data generated after modifying a first file and metadata corresponding to the first file;

[0158] A determining unit 1202 is configured to determine second identification group information of second snapshot data stored in the metadata;

[0159] The deleting unit 1203 is used to search for third snapshot data corresponding to the third identification group information from the second snapshot data and delete the third snapshot data when third identification group information other than the first identification group information is found from the second identification group information.

[0160] In one embodiment, the deletion unit 1203 is also used to obtain fourth snapshot data; the fourth snapshot data is a sub-snapshot of the third snapshot data; determine whether the fourth snapshot data has a sub-snapshot dependency; if the fourth snapshot data does not have the sub-snapshot dependency, delete the third snapshot data; if the fourth snapshot data has the sub-snapshot dependency, search the third snapshot data for the first part of snapshot data that does not have the sub-snapshot dependency, and delete the first part of snapshot data.

[0161] In one embodiment, the deletion unit 1203 is also used to obtain at least one tag information of the fourth snapshot data; when each of the tag information in the at least one tag information is the same as the preset information, it is determined that the fourth snapshot data does not have the sub-snapshot dependency; when the first tag information in the at least one tag information is different from the preset information, it is determined that the fourth snapshot data has the sub-snapshot dependency.

[0162] In one embodiment, the deletion unit 1203 is also used to mark the third snapshot data when it is determined that the fourth snapshot data has the sub-snapshot dependency; when an update operation on the first file is detected, the first marking information is modified to the preset information, the second part of the snapshot data corresponding to the first marking information in the third snapshot data is deleted, and the marking of the third snapshot data is unmarked.

[0163] In one embodiment, the data processing device 1200 also includes a creation unit, which is used to search for fifth snapshot data corresponding to the fourth identification group information from the first snapshot data and create the fifth snapshot data when fourth identification group information other than the second identification group information is found from the first identification group information.

[0164] In one embodiment, the creation unit is also used to fill at least one mark information of the fourth snapshot data into a blank mark table corresponding to the fifth snapshot data to obtain second mark information corresponding to the fifth snapshot data; clear the at least one mark information, and store the second mark information in the metadata corresponding to the fifth snapshot data to create the fifth snapshot data.

[0165] It should be noted that: when the data processing device provided in the above embodiment performs data processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0166] Based on the hardware implementation of the above-mentioned program module, an embodiment of the present application also provides a data processing device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the data processing method provided in the above-mentioned embodiment are implemented.

[0167] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the above embodiment are implemented.

[0168] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0169] It should be noted that Fig.13 Schematic diagram of a hardware entity structure of a data processing device in an embodiment of the present application; Fig.13 As shown, the hardware entity of the electronic device 1300 includes: a processor 1301 and a memory 1303 . Optionally, the electronic device 1300 may also include a communication interface 1302 .

[0170] Optionally, the electronic device 1300 may further include a bus, which is a communication bus, for implementing connection and communication between the processor 1301 and the memory 1303 .

[0171] It can be understood that the memory 1303 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 1303 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0172] The method disclosed in the above embodiment of the present application can be applied to the processor 1301, or implemented by the processor 1301. The processor 1301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 1301 or an instruction in the form of software. The above processor 1301 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1301 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 1303, and the processor 1301 reads the information in the memory 1303 and completes the steps of the above method in combination with its hardware.

[0173] The specific steps are as follows: the processor 1301 is used to obtain the first identification group information of the first snapshot data generated after modifying the first file and the metadata corresponding to the first file; determine the second identification group information of the second snapshot data stored in the metadata; when the third identification group information other than the first identification group information is found from the second identification group information, search for the third snapshot data corresponding to the third identification group information from the second snapshot data, and delete the third snapshot data.

[0174] In one embodiment, the processor 1301 is also used to obtain fourth snapshot data; the fourth snapshot data is a sub-snapshot of the third snapshot data; determine whether the fourth snapshot data has a sub-snapshot dependency; if the fourth snapshot data does not have the sub-snapshot dependency, delete the third snapshot data; if the fourth snapshot data has the sub-snapshot dependency, search the third snapshot data for the first part of snapshot data that does not have the sub-snapshot dependency, and delete the first part of snapshot data.

[0175] In one embodiment, the processor 1301 is further used to obtain at least one tag information of the fourth snapshot data; when each of the tag information in the at least one tag information is the same as the preset information, it is determined that the fourth snapshot data does not have the sub-snapshot dependency; when the first tag information in the at least one tag information is different from the preset information, it is determined that the fourth snapshot data has the sub-snapshot dependency.

[0176] In one embodiment, the processor 1301 is further used to mark the third snapshot data when it is determined that the fourth snapshot data has the sub-snapshot dependency; when an update operation on the first file is detected, the first marking information is modified to the preset information, the second part of the snapshot data corresponding to the first marking information in the third snapshot data is deleted, and the marking of the third snapshot data is unmarked.

[0177] In one embodiment, the processor 1301 is further used to search for fifth snapshot data corresponding to the fourth identification group information from the first snapshot data and create the fifth snapshot data when fourth identification group information other than the second identification group information is found from the first identification group information.

[0178] In one embodiment, the processor 1301 is also used to fill at least one mark information of the fourth snapshot data into a blank mark table corresponding to the fifth snapshot data to obtain second mark information corresponding to the fifth snapshot data; clear the at least one mark information, and store the second mark information in the metadata corresponding to the fifth snapshot data to create the fifth snapshot data.

[0179] In an exemplary embodiment, the device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to perform the aforementioned method.

[0180] Illustratively, an embodiment of the present application further provides a computer program product, including a computer program; the computer program can be executed by the processor 1301 of the electronic device 1300 to complete the steps of any of the aforementioned methods.

[0181] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0182] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0183] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0184] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0185] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0186] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0187] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire first identification group information of first snapshot data generated after modifying the first file and metadata corresponding to the first file; Determine second identification group information of second snapshot data stored in the metadata; In the case where a third identification group information other than the first identification group information is found in the second identification group information, searching the second snapshot data for third snapshot data corresponding to the third identification group information, and deleting the third snapshot data; Wherein, deleting the third snapshot data includes: Acquire fourth snapshot data; the fourth snapshot data is a sub-snapshot of the third snapshot data; Determine whether the fourth snapshot data has a sub-snapshot dependency; If the fourth snapshot data does not have the sub-snapshot dependency, deleting the third snapshot data; In the case that the fourth snapshot data has the sub-snapshot dependency, the first portion of snapshot data that does not have the sub-snapshot dependency is searched from the third snapshot data, and the first portion of snapshot data is deleted.

2. The method according to claim 1, characterized in that The determining whether the fourth snapshot data has a sub-snapshot dependency includes: Acquire at least one tag information of the fourth snapshot data; In a case where each of the marking information in the at least one marking information is the same as the preset information, determining that the fourth snapshot data has no sub-snapshot dependency; When the first marking information in the at least one marking information is different from the preset information, it is determined that the fourth snapshot data has the sub-snapshot dependency.

3. The method according to claim 2, characterized in that The method further comprises: In the case where it is determined that the fourth snapshot data has the sub-snapshot dependency, marking the third snapshot data; When an update operation on the first file is detected, the first marking information is modified to the preset information, the second part of the snapshot data corresponding to the first marking information in the third snapshot data is deleted, and the marking of the third snapshot data is removed.

4. The method according to claim 1, characterized in that The method further comprises: When fourth identification group information other than the second identification group information is found in the first identification group information, fifth snapshot data corresponding to the fourth identification group information is found in the first snapshot data, and the fifth snapshot data is created.

5. The method according to claim 4, characterized in that The step of creating the fifth snapshot data includes: Filling at least one tag information of the fourth snapshot data into a blank tag table corresponding to the fifth snapshot data to obtain second tag information corresponding to the fifth snapshot data; The at least one mark information is cleared, and the second mark information is stored in metadata corresponding to the fifth snapshot data to create the fifth snapshot data.

6. A data processing device, characterized in that: The device comprises: An acquiring unit, configured to acquire first identification group information of first snapshot data generated after modifying the first file and metadata corresponding to the first file; a determining unit, configured to determine second identification group information of second snapshot data stored in the metadata; a deleting unit, configured to, when finding a third identification group information other than the first identification group information in the second identification group information, search the second snapshot data for third snapshot data corresponding to the third identification group information, and delete the third snapshot data; The deleting unit is further used to obtain fourth snapshot data; the fourth snapshot data is a sub-snapshot of the third snapshot data; determine whether the fourth snapshot data has a sub-snapshot dependency; if the fourth snapshot data does not have the sub-snapshot dependency, delete the third snapshot data; if the fourth snapshot data has the sub-snapshot dependency, search the third snapshot data for the first part of snapshot data that does not have the sub-snapshot dependency and delete the first part of snapshot data.

7. A data processing device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that The computer program implements the steps of the method according to any one of claims 1 to 5 when executed by a processor.

9. A storage medium having a computer program stored thereon, characterized in that: The computer program implements the steps of the method according to any one of claims 1 to 5 when executed by a processor.

Images