A data processing method, device, equipment and storage medium

Abstract

The present application provides a data processing method, device, equipment and storage medium, which relates to the field of data protection and aims to solve the problem of data being unable to be restored due to accidental deletion of data when performing backup, recovery and garbage collection concurrently. The method includes: in the current round of garbage collection process, all available data fingerprints that are filtered out and belong to expired snapshots are converted into forgettable data fingerprints, and all backup processes that occurred during the current round of garbage collection process are recorded; in the next round of garbage collection process, wait until all recorded backup processes have ended, check whether any of the recorded backup processes has used the forgettable data fingerprint, and if a forgettable data fingerprint has been used by one or more recorded backup processes, convert the forgettable data fingerprint into an available data fingerprint; in the recovery process, use the available data fingerprint and the forgettable data fingerprint to restore the snapshot of the data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data protection, and specifically, to a data processing method, apparatus, device and storage medium. Background Art

[0002] In the backup and recovery system, the data in the backup library needs to be scanned and sorted regularly to clean up expired or duplicate data, that is, garbage collection. When the backup process and garbage collection process are processed concurrently, some metadata and data will be processed at the same time at a certain point in time, and data processing conflicts will occur, resulting in accidental deletion of data, making data unrecoverable and the backup and recovery system unavailable. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, apparatus, device and storage medium, which are intended to solve the problem of accidental deletion of data and irrecoverable data when backup, recovery and garbage collection are performed concurrently.

[0004] A first aspect of an embodiment of the present application provides a data processing method, including: In the backup process, each data block to be backed up is processed to obtain a data fingerprint of each data block to be backed up; Compare the data fingerprint of each data block to be backed up with each data fingerprint in the data fingerprint library; When the data fingerprint of a data block to be backed up is different from all the data fingerprints in the data fingerprint library, the data fingerprint of the data block to be backed up is converted into an available data fingerprint and stored in the data fingerprint library, and the data block to be backed up is stored in the pack layer; In the current round of garbage collection process, all available class data fingerprints unique to expired snapshots are filtered out from available class data fingerprints, and all filtered available class data fingerprints are converted into forgettable class data fingerprints, and all backup processes occurring during the current round of garbage collection process are recorded; In the next round of garbage collection process, wait until all the recorded backup processes have ended, check whether any of the recorded backup processes has used the forgettable data fingerprint, and if a forgettable data fingerprint has been used by one or more recorded backup processes, convert the forgettable data fingerprint into an available data fingerprint; In the recovery process, the available data fingerprint and the forgettable data fingerprint are used to recover a snapshot of the data.

[0005] In an optional embodiment, the method further includes: When a data fingerprint of a data block to be backed up is identical to a data fingerprint of the data fingerprint library, it is determined that the data block to be backed up is a data block that has been backed up.

[0006] In an optional embodiment, the method further includes: When a forgettable data fingerprint has not been used by all the recorded backup processes, the forgettable data fingerprint is converted into a deletable data fingerprint; The deletable data fingerprint is deleted from the data fingerprint library, and the data block corresponding to the deletable data fingerprint is marked as deletable in the pack layer.

[0007] In an optional implementation manner, deleting the deletable data fingerprint from the data fingerprint library includes: Move the deletable data fingerprint to the local recycle bin; After the first delay period, check whether the deletable data fingerprint has been used by any backup process that occurred within the first delay period; When the deletable data fingerprint has not been used by any backup process occurring within the first period of time, the deletable data fingerprint is deleted from the local recycle bin.

[0008] In an optional implementation manner, marking the data block corresponding to the deletable data fingerprint as deletable in the pack layer includes: Record the data blocks corresponding to the deletable data fingerprints into the local recycle bin; If it is determined multiple times that the data block corresponding to the deletable data fingerprint has not been used by any backup process occurring within the first time period, the data block corresponding to the deletable data fingerprint is marked as deletable in a pack file, wherein the pack layer includes multiple pack files; When all data blocks in the pack file are not used by any backup process, the pack file is deleted from the local recycle bin; When the data block corresponding to the deletable data fingerprint has been used by one or more backup processes occurring within the first time period, the record of the data block corresponding to the deletable data fingerprint in the local recycle bin is revoked, and the deletable data fingerprint is converted into an available data fingerprint.

[0009] In an optional embodiment, the method further includes: Create a garbage backup repository for multiple user backup repositories belonging to the same region; Before performing garbage collection on the local recycle bin in each user backup repository, copy the garbage data in the local recycle bin in the user backup repository to the garbage backup repository; Delete the junk data in the local recycle bin in the user's backup library; The garbage data in the garbage backup library is deleted after a delay of a second time period, and the second time period is greater than the first time period.

[0010] In an optional embodiment, the method further includes: Before the second time period expires, when an abnormality occurs in a local recycle bin in a user backup repository, the garbage data in the local recycle bin in the user backup repository is restored using the garbage data in the garbage backup repository.

[0011] A second aspect of an embodiment of the present application provides a data processing device, including: A first processing module is used to process each data block to be backed up in the backup process to obtain a data fingerprint of each data block to be backed up; A comparison module, used for comparing the data fingerprint of each data block to be backed up with each data fingerprint in the data fingerprint library; The second processing module is used for, when the data fingerprint of a data block to be backed up is different from each data fingerprint in the data fingerprint library, converting the data fingerprint of the data block to be backed up into a usable data fingerprint and storing it in the data fingerprint library, and storing the data block to be backed up in the pack layer; The third processing module is used to filter out all available class data fingerprints unique to expired snapshots from available class data fingerprints in the current round of garbage collection process, convert all filtered available class data fingerprints into forgettable class data fingerprints, and record all backup processes occurring during the current round of garbage collection process; The fourth processing module is used to wait for all the recorded backup processes to be completed in the next round of garbage collection process, check whether any of the recorded backup processes has used the forgettable data fingerprint, and if a forgettable data fingerprint has been used by one or more recorded backup processes, convert the forgettable data fingerprint into an available data fingerprint; The recovery module is used to recover a snapshot of the data using the usable data fingerprint and the forgettable data fingerprint.

[0012] A third aspect of an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method of the first aspect of the embodiment of the present application when executing the computer program.

[0013] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data processing method of the first aspect of the embodiment of the present application is implemented.

[0014] In this embodiment, in the backup process, each data block to be backed up is processed to obtain a data fingerprint of each data block to be backed up; the data fingerprint of each data block to be backed up is compared with each data fingerprint of the data fingerprint library; when the data fingerprint of a data block to be backed up is different from each data fingerprint of the data fingerprint library, the data fingerprint of the data block to be backed up is converted into an available class data fingerprint and stored in the data fingerprint library, and the data block to be backed up is stored in the pack layer; in the current round of garbage collection process, all available class data fingerprints unique to expired snapshots are filtered out from the available class data fingerprints, and all filtered available class data fingerprints are converted into forgettable class data fingerprints, and all backup processes occurring during the current round of garbage collection process are recorded; in the next round of garbage collection process, wait until all recorded backup processes have ended, check whether any recorded backup process has used the forgettable class data fingerprint, and when a forgettable class data fingerprint has been used by one or more recorded backup processes, convert the forgettable class data fingerprint into an available class data fingerprint; in the recovery process, the available class data fingerprint and the forgettable class data fingerprint are used to restore the snapshot of the data.

[0015] The specific beneficial effects are: by classifying data fingerprints (available data fingerprints, forgettable data fingerprints), delaying data deletion, avoiding conflicts between backup and recovery tasks and garbage collection in processing the same metadata and data, and achieving concurrent execution of backup and recovery tasks and garbage collection in a lock-free state, thereby improving the stability of backup and recovery tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 This is a data management component architecture diagram of a backup and recovery system of the present application; Figure 2 This is a relationship diagram of fp / index and pack in this application; Figure 3 is a flowchart of the steps of a data processing method proposed in one embodiment of the present application; Figure 4 This is a schematic diagram of a data fingerprint classification delayed deletion strategy of a data processing method proposed in an embodiment of the present application; Figure 5 It is a schematic diagram of a delayed deletion strategy of a local recycling bin in a data processing method proposed in an embodiment of the present application; Figure 6 It is a schematic diagram of a delayed deletion strategy of a garbage backup library of a data processing method proposed in an embodiment of the present application; Figure 7 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] Data fingerprint (FP): An identifier or signature generated by processing specified data using a general or custom hash function, used to identify or distinguish data.

[0020] Data slicing (Chunk): In the backup and recovery system, especially the file system backup, the file or data stream is divided into fixed-size or variable-size data blocks, and the fingerprint is generated by the hash function. The fingerprint library is compared and updated to quickly realize backup and deduplication, reducing backup time and storage space.

[0021] Snapshot recycling GC (garbage collection): Automatically recycles expired backup snapshot data, including data and metadata, based on the user's settings for the snapshot retention period.

[0022] Backup snapshot: In the backup and recovery system, according to the user's backup strategy, the data that the user needs to protect is periodically copied and stored at a certain point in time or a certain period of time, and finally a physical or logical copy of the original data is generated, so that the data can be quickly restored based on the snapshot copy after data loss or malicious damage.

[0023] In today's information world, whether it is the government, enterprises or individuals, a large amount of data will be generated and obtained in production and life. Data has become one of the core assets of enterprises and organizations. It is extremely important to ensure the security and availability of core and key data. Data backup is an important means of data protection, which is mainly reflected in the following aspects: Data ransomware: data is attacked by viruses, ransomware, etc., encrypted, destroyed or tampered with; hardware failure: disk damage or aging makes the stored data unavailable; misoperation: users or software defects cause accidental deletion, modification, etc., resulting in accidental loss or damage of data; compliance requirements: some industries, such as finance, medical care, government and enterprises are subject to laws and regulations and need to back up data regularly to meet audit and regulatory requirements.

[0024] The design of common backup and recovery systems is also a classic C / S architecture, as follows: Client: Also known as the backup agent, responsible for reading data. According to the actual application scenario and requirements, it performs additional processing on the read data, such as slicing data chunks, comparing the fingerprint library for source-side deduplication, etc. Finally, the data needs to be transferred to the Server for further processing and protection. Server: Responsible for slicing data, or comparing the chunks that have been sliced ​​by the client with the fp fingerprint library. For non-duplicate data, it is necessary to update the fingerprint library and save the data part of the sliced ​​chunk; for duplicate data, it is only necessary to update the metadata such as the fingerprint library to achieve data deduplication and data integrity.

[0025] In the backup and recovery system, different users or the same user will usually formulate different retention policies for different data, usually with a retention period of hours, days, months, years or forever, that is, the backup snapshots will have different life cycles in the backup library according to the protection level of the backup data. Therefore, the data in the backup library needs to be scanned and sorted regularly to clean up expired or duplicate data, that is, GC garbage collection.

[0026] GC garbage collection is relatively complex and has a large impact in the backup and recovery system, which is reflected in the following two aspects: First, the difficulty of GC technology. Data between snapshots is shared, and snapshot deletion requires the stripping and deletion of exclusive data; GC cannot affect the execution of backup and recovery, so under concurrent conditions, it is necessary to quickly and accurately eliminate the re-reference of backup and recovery tasks. Second, the impact of GC on the backup system. Accidental deletion of data: It will cause the backup data to be unable to be restored, violate the external service commitment, cause compliance risks and potential economic losses; GC is too slow: If garbage collection cannot be cleaned up in time, it will lead to tight storage resources in the backup library, increase user storage costs, and the fingerprint library has a large number of junk fingerprints, which will affect the speed of fingerprint queries and the performance of backup and recovery.

[0027] When the backup and GC processes are processed concurrently, some metadata and data will be processed simultaneously. At a certain point in time, the data will be processed simultaneously. For example, GC believes that a fingerprint is only referenced in an expired snapshot and is not referenced in all available backup snapshots, but the fingerprint library has not been updated to be unavailable. At this time, in the backup process, the fingerprint of the slice is queried, and only the metadata is updated, and the data part of the chunk is not saved. If GC directly deletes junk data, including the chunk's fingerprint fp and data data, the conflict in data processing will eventually lead to accidental deletion of data, making the data unrecoverable and the backup and recovery system unavailable.

[0028] Introducing distributed locks or other lock services can solve the conflict between GC and backup data processing to a certain extent. However, this solution limits the timing of backup and cannot meet the execution of the periodic backup strategy specified by the customer. It has great limitations and cannot meet the needs of enterprise-level backup and recovery.

[0029] like Figure 1 As shown in the figure, in common backup and recovery systems, data management is generally divided into three layers: snapshot layer, fp fingerprint library and slice data pack part. The specific design and functions are as follows: The two parts of a chunk, fp and data, are stored in the fingerprint library and the pack of the chunk respectively. fp is generally the hash value of sha256, and the data of multiple chunks are assembled into a pack file for storage.

[0030] The data part of the chunk identifies the storage object in the pack according to the index, that is, the pack file name, offset and size. For a specific diagram, please refer to Figure 2 .

[0031] The chunk's fingerprint fp can be indexed to a specific index in the fingerprint library, and then indexed to the data part of the chunk in the pack layer.

[0032] Each snapshot of the snapshot layer can completely restore the snapshot of the data based on the fingerprint library and the pack layer.

[0033] exist Figure 1 In the process, if the backup and GC access and update the fingerprint library at the same time, it will cause resource usage conflicts, resulting in inconsistent index or pack data, and ultimately lead to backup library data corruption and snapshot unavailability.

[0034] In view of the above problems, the embodiments of the present application propose a data processing method, device, equipment and storage medium to improve the reliability of the backup and recovery system. The following is a detailed description of a data processing method, device, equipment and storage medium provided by the embodiments of the present application through some embodiments and their application scenarios in conjunction with the accompanying drawings.

[0035] Reference Figure 3 , Figure 3 1 is a flowchart of the steps of the data processing method proposed in one embodiment of the present application. Figure 3 As shown, the method includes the following steps S11 to S16: Step S11: In the backup process, each data block to be backed up is processed to obtain a data fingerprint of each data block to be backed up.

[0036] In the backup process, a general or custom hash function is used to process each data block (data slice) to be backed up, thereby generating a data fingerprint for each data block to be backed up. The data block to be backed up is the data block of fixed or variable size that the backup file or data stream is divided into; the data fingerprint is an identifier or signature generated by processing the specified data, which is used to identify or distinguish the data.

[0037] Step S12: Compare the data fingerprint of each data block to be backed up with each data fingerprint in the data fingerprint library.

[0038] The data fingerprint of each data block to be backed up is compared with the existing data fingerprints in the data fingerprint library to check whether the current data block to be backed up has been stored in the previous backup. The data fingerprint library is a database that stores all data fingerprints generated during the previous backup process.

[0039] Step S13: When the data fingerprint of a data block to be backed up is different from all the data fingerprints in the data fingerprint library, the data fingerprint of the data block to be backed up is converted into an available data fingerprint and stored in the data fingerprint library, and the data block to be backed up is stored in the pack layer.

[0040] If the data fingerprint of a data block to be backed up is different from all the data fingerprints in the data fingerprint library, it means that this is a new data block that has not been backed up before. The data fingerprint is converted into an available data fingerprint and stored in the data fingerprint library, and the corresponding data block is stored in the pack layer, which is a physical or logical structure for storing data blocks.

[0041] Furthermore, when a data fingerprint of a data block to be backed up is identical to a data fingerprint of the data fingerprint library, it is determined that the data block to be backed up is a data block that has been backed up.

[0042] If the data fingerprint of a data block to be backed up is the same as a data fingerprint in the data fingerprint library, it means that the data block has been backed up before, and there is no need to store the data block to be backed up again.

[0043] Step S14: In the current round of garbage collection process, all available class data fingerprints unique to expired snapshots are filtered out from the available class data fingerprints, and all filtered available class data fingerprints are converted into forgettable class data fingerprints, and all backup processes occurring during the current round of garbage collection process are recorded.

[0044] In the current garbage collection process, filter out all available class data fingerprints that are unique to expired snapshots (exceeding the set retention period) from available class data fingerprints, and convert all filtered available class data fingerprints into forgettable class data fingerprints. Record all backup processes that occurred during the current garbage collection process. By converting the available class data fingerprints of expired snapshots into forgettable class data fingerprints, data deletion is delayed to avoid data processing conflicts when performing backup and garbage collection concurrently.

[0045] Step S15: In the next round of garbage collection process, wait until all recorded backup processes have ended, check whether any of the recorded backup processes has used the forgettable data fingerprint, and if a forgettable data fingerprint has been used by one or more recorded backup processes, convert the forgettable data fingerprint into an available data fingerprint.

[0046] In the next round of garbage collection, after all the recorded backup processes have ended, check whether these backup processes have used the forgettable data fingerprints. If a forgettable data fingerprint has been used by one or more recorded backup processes, it means that the forgettable data fingerprint is useful, so it is converted to a usable data fingerprint. This helps ensure that only data fingerprints and data blocks that are no longer needed are deleted to avoid accidental deletion of data.

[0047] Furthermore, the method further comprises: When a forgettable data fingerprint has not been used by all the recorded backup processes, the forgettable data fingerprint is converted into a deletable data fingerprint; The deletable data fingerprint is deleted from the data fingerprint library, and the data block corresponding to the deletable data fingerprint is marked as deletable in the pack layer.

[0048] If a forgettable data fingerprint is detected during the garbage collection process as not used by any recorded backup process (i.e., it is not referenced by any valid backup snapshot), it means that the forgettable data fingerprint and the corresponding data block are not needed, and the forgettable data fingerprint is converted into a deletable data fingerprint. The deletable data fingerprint is deleted from the data fingerprint library, and the data block corresponding to the deletable data fingerprint is marked as deletable in the pack layer.

[0049] like Figure 4 As shown in the figure, by dividing the data fingerprint, that is, the index, into three categories: active, forgettable, and delete, the mutual influence of backup and GC can be isolated, and the conflict between the job being backed up and restored and the GC processing the same metadata and data in the same time period is resolved. The concurrent execution of backup and recovery tasks and GC in a lock-free state is achieved, which can avoid accidental deletion of data and improve the stability of backup and recovery tasks.

[0050] Available class active: New backup snapshots, compared fingerprint libraries, new indexes, that is, data fingerprints will be updated to this class and become available class data fingerprints.

[0051] Forgettable class forget: Only the GC process will reference, process and update this class. In the current round of GC, according to the expired snapshot, filter the unique index part from the available class active and transfer it to this class (convert the filtered available class data fingerprint to the forgettable class data fingerprint), and remember the timestamp of the transfer, or all backup tasks at that time; in the next round of GC, if all the backup tasks saved in the forgettable class forget of the previous round of GC have ended, then in one or more backup snapshots at that time, whether the index in the forgettable class forget is referenced (whether the forgettable class data fingerprint has been used), if there is no reference, transfer to the deletable class delete (convert the unused forgettable class data fingerprint to the deletable class data fingerprint); if there is a reference, transfer to the available class active (convert the used forgettable class data fingerprint to the available class data fingerprint).

[0052] Deletable class delete: Only the GC process will reference, process and update this class. The indexes under this class (deletable class data fingerprints) are indexes that are not referenced by any available snapshots and can be safely deleted, and the data part of the chunk indexed by the index can also be deleted.

[0053] Considering other abnormal situations or bugs in software coding, there is still a theoretical possibility of accidental deletion, so the local recycle bin recycle strategy is introduced. For any backed up files or objects, when deleting, according to different file or object types, they are first moved or recorded to the recycle, and then according to the deletion strategy or algorithm, after a delay or process, the deletion process is carried out after confirmation that they can be deleted.

[0054] Furthermore, deleting the deletable data fingerprint from the data fingerprint library includes: Move the deletable data fingerprint to the local recycle bin; After the first delay period, check whether the deletable data fingerprint has been used by any backup process that occurred within the first delay period; When the deletable data fingerprint has not been used by any backup process occurring within the first period of time, the deletable data fingerprint is deleted from the local recycle bin.

[0055] For data fingerprints, that is, metadata files such as indexes, which are generally small in size and few in number, delayed deletion means moving them directly to the local recycle bin. According to the retention time TTL, such as 3 days, after 3 days, the data integrity is verified and the data can be safely deleted after no data is deleted by mistake.

[0056] Specifically, Figure 5 As shown, the deletable data fingerprints are moved from the data fingerprint library to the local recycle bin. According to the deletion policy or algorithm, after a delay of the first period (retention period, such as 3 days), each deletable data fingerprint in the recycle bin is checked again to verify whether they have been used by any backup process during this period. If a data fingerprint has not been used within the first period, it means that it is no longer needed and can be deleted from the recycle bin; if a data fingerprint has been used by one or more backup processes before the end of the first period, it means that it is still needed and can be restored from the recycle bin and the data fingerprint library is updated. Delayed deletion based on the first period can reduce the risk of accidental data deletion to ensure the accuracy and reliability of data deletion.

[0057] Furthermore, marking the data block corresponding to the deletable data fingerprint as deletable in the pack layer includes: Record the data blocks corresponding to the deletable data fingerprints into the local recycle bin; If it is determined multiple times that the data block corresponding to the deletable data fingerprint has not been used by any backup process occurring within the first time period, the data block corresponding to the deletable data fingerprint is marked as deletable in a pack file, wherein the pack layer includes multiple pack files; When all data blocks in the pack file are not used by any backup process, the pack file is deleted from the local recycle bin; When the data block corresponding to the deletable data fingerprint has been used by one or more backup processes occurring within the first time period, the record of the data block corresponding to the deletable data fingerprint in the local recycle bin is revoked, and the deletable data fingerprint is converted into an available data fingerprint.

[0058] For pack files of data blocks, such files or objects are generally large in size and large in number. Delayed deletion is only recorded in the record file under the path of the local recycle bin. After several rounds of GC confirmation (configurable according to actual conditions, such as 3 confirmations), all data blocks in the pack file are not referenced by any index and can be safely deleted.

[0059] Specifically, Figure 5 As shown, the data block corresponding to the deletable data fingerprint is recorded in the record file in the local recycle bin. When it is determined multiple times that the data block corresponding to the deletable data fingerprint has not been used by any backup process that occurred within the first time period, it means that it is no longer needed. The data block corresponding to the deletable data fingerprint is marked as deletable in the pack file to which it belongs. There are multiple pack files in the pack layer. When all the data blocks in the pack file are not used by any backup process, the pack file is deleted from the local recycle bin. When the data block corresponding to the deletable data fingerprint is used by one or more backup processes that occurred within the first time period, it means that it is still needed. The record of the data block corresponding to the deletable data fingerprint in the local recycle bin is revoked, and the deletable data fingerprint is converted into an available data fingerprint. The multiple confirmation mechanism of the data file provides the function and means of GC self-checking. Once the recorded deletable file is found to be referenced again, it means that the data has been deleted by mistake. At this time, it is necessary to stop GC, eliminate and repair the problem, and avoid the accidental deletion of data.

[0060] Furthermore, the method further comprises: Create a garbage backup repository for multiple user backup repositories belonging to the same region; Before performing garbage collection on the local recycle bin in each user backup repository, copy the garbage data in the local recycle bin in the user backup repository to the garbage backup repository; Delete the junk data in the local recycle bin in the user's backup library; The garbage data in the garbage backup library is deleted after a delay of a second time period, and the second time period is greater than the first time period.

[0061] like Figure 6As shown, for the application scenario of the cloud backup and recovery system, a trash backup library (trash bucket) is established for the backup libraries of all users in the same area. The main function of the trash backup library is to temporarily store the garbage data (that is, data marked as deleted or no longer needed) cleaned up from the local recycle bin in each user's backup library. Before garbage collection is performed on the local recycle bin in each user's backup library, the garbage data in the local recycle bin in the user's backup library is copied to the trash backup library, and the garbage data in the local recycle bin in the user's backup library is deleted. For the garbage data in the trash backup library, the deletion is delayed for a second period of time (storage time, such as 30 days). The second period of time can be set according to the function of the object storage, and the second period of time is longer than the first period of time.

[0062] Furthermore, the method further comprises: Before the second time period expires, when an abnormality occurs in a local recycle bin in a user backup repository, the garbage data in the local recycle bin in the user backup repository is restored using the garbage data in the garbage backup repository.

[0063] Before the end of the second period, if an abnormal situation occurs in the local recycle bin of a user's backup library (data is deleted or lost by mistake), the garbage data in the garbage backup library can be used to restore the garbage data in the local recycle bin of the user's backup library. By creating a garbage backup library, the safe management and possible recovery of garbage data can be achieved, which helps to balance the effective use of storage space and the need for data security.

[0064] Step S16: In the recovery process, the available data fingerprint and the forgettable data fingerprint are used to recover a snapshot of the data.

[0065] In the recovery process, the available data fingerprint and the forgettable data fingerprint can be used to restore the data snapshot. By finding the data fingerprint associated with the required snapshot and retrieving the corresponding data block from the stored pack layer, the data snapshot can be completely restored.

[0066] In the backup and recovery system, a backup library generally supports multiple backup policies configured by different users. Different backup policies can set different backup snapshot retention periods, from hourly level to daily, weekly, monthly, annual, or even permanent retention. That is, different snapshots have different life cycles, and different snapshots share a lot of data and metadata. Therefore, the recovery of different snapshots requires stripping and deleting the exclusive data of each snapshot, and excluding the impact of the backup and recovery being executed. This is technically difficult to achieve and has a large impact.

[0067] Compared with traditional or conventional snapshot recycling GC technology, the delayed deletion of this application can effectively reduce the risk of accidental data deletion during the GC process, and can achieve data loss and reconstruction in the future, while solving timing problems such as concurrent execution of backup and recovery, simplifying the GC process and improving reliability. The specific advantages are as follows: By dividing data fingerprints, namely indexes, into three categories: active, forgettable, and delete, we can isolate the mutual impact of backup and GC, and solve the conflict between the jobs being backed up and restored and the GC processing the same metadata and data in the same time period, thus achieving concurrent execution of backup and restoration tasks and GC in a lock-free state.

[0068] By using the local recycle bin recycle and deleting data based on the retention time TTL (first time length) and the multiple confirmation mechanism of data files, the risk of accidental data deletion can be reduced to ensure the accuracy and reliability of data deletion.

[0069] By using a garbage backup library, we ensure that there is a safe copy of the garbage data in the local recycle bins of multiple users before deletion, and set a second time period to allow recovery operations before the data is finally deleted. In the event of accidental deletion of data, a means of data repair is provided to further reduce the risk of accidental deletion of data.

[0070] It is simple and easy to implement. For backup and recovery systems that do not implement delayed deletion, whether they are traditional or cloud-based backup and recovery systems, generally, as long as the deletion operation process is encapsulated, the delayed deletion of the application can be seamlessly implemented, with minor changes and greater benefits.

[0071] Based on the same inventive concept, an embodiment of the present application provides a data processing device, the device comprising: A first processing module is used to process each data block to be backed up in the backup process to obtain a data fingerprint of each data block to be backed up; A comparison module, used for comparing the data fingerprint of each data block to be backed up with each data fingerprint in the data fingerprint library; The second processing module is used for, when the data fingerprint of a data block to be backed up is different from each data fingerprint in the data fingerprint library, converting the data fingerprint of the data block to be backed up into a usable data fingerprint and storing it in the data fingerprint library, and storing the data block to be backed up in the pack layer; The third processing module is used to filter out all available class data fingerprints unique to expired snapshots from available class data fingerprints in the current round of garbage collection process, convert all filtered available class data fingerprints into forgettable class data fingerprints, and record all backup processes occurring during the current round of garbage collection process; The fourth processing module is used to wait for all the recorded backup processes to be completed in the next round of garbage collection process, check whether any of the recorded backup processes has used the forgettable data fingerprint, and if a forgettable data fingerprint has been used by one or more recorded backup processes, convert the forgettable data fingerprint into an available data fingerprint; The recovery module is used to recover a snapshot of the data using the usable data fingerprint and the forgettable data fingerprint.

[0072] In an optional embodiment, the device further comprises: The determination module is used to determine that the data block to be backed up is a data block that has been backed up when the data fingerprint of the data block to be backed up is the same as a data fingerprint of the data fingerprint library.

[0073] In an optional embodiment, the device further comprises: A fifth processing module, configured to convert a forgettable data fingerprint into a deletable data fingerprint when the forgettable data fingerprint has not been used by all the recorded backup processes; The deletion module is used to delete the deletable data fingerprint from the data fingerprint library, and mark the data block corresponding to the deletable data fingerprint as deletable in the pack layer.

[0074] In an optional implementation, the deletion module includes: The first deletion sub-unit is used to move the deletable data fingerprint to the local recycle bin; after a first delay, check whether the deletable data fingerprint has been used by any backup process that occurred within the first time period; if the deletable data fingerprint has not been used by any backup process that occurred within the first time period, delete the deletable data fingerprint from the local recycle bin.

[0075] In an optional implementation, the deletion module includes: The second deletion sub-unit is used to record the data block corresponding to the deletable data fingerprint into the local recycle bin; when it is determined for multiple times that the data block corresponding to the deletable data fingerprint has not been used by any backup process occurring within the first time period, mark the data block corresponding to the deletable data fingerprint as deletable in the pack file, and the pack layer includes multiple pack files; when all the data blocks in the pack file are not used by any backup process, delete the pack file from the local recycle bin; when the data block corresponding to the deletable data fingerprint has been used by one or more backup processes occurring within the first time period, cancel the record of the data block corresponding to the deletable data fingerprint in the local recycle bin, and convert the deletable data fingerprint into an available data fingerprint.

[0076] In an optional embodiment, the device further comprises: A creation module for creating a garbage backup repository for multiple user backup repositories belonging to the same region; A copy module, used for copying the garbage data in the local recycle bin in each user backup repository to the garbage backup repository before performing garbage collection on the local recycle bin in the user backup repository; A sixth processing module, used for deleting the garbage data in the local recycle bin in the backup library of the user; The seventh processing module is used to delete the garbage data in the garbage backup library after a second delay, where the second delay is greater than the first delay.

[0077] In an optional embodiment, the device further comprises: The eighth processing module is used to use the garbage data in the garbage backup library to restore the garbage data in the local recycle bin in the user backup library when an abnormality occurs in the local recycle bin in the user backup library before the second time period expires.

[0078] Based on the same inventive concept, an embodiment of the present application provides an electronic device, referring to Figure 7 , Figure 7 FIG. 1 is a schematic diagram of an electronic device shown in an embodiment of the present application. Figure 7 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the data processing method of the embodiment of the present disclosure.

[0079] Based on the same inventive concept, an embodiment of the present disclosure also provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of a computer device, the computer device can execute the steps in the data processing method described in any of the above embodiments of the present application.

[0080] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0082] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0086] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0087] The above is a detailed introduction to a data processing method, device, equipment and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: include: In the backup process, each data block to be backed up is processed to obtain a data fingerprint of each data block to be backed up; Compare the data fingerprint of each data block to be backed up with each data fingerprint in the data fingerprint library; When the data fingerprint of a data block to be backed up is different from all the data fingerprints in the data fingerprint library, the data fingerprint of the data block to be backed up is converted into an available data fingerprint and stored in the data fingerprint library, and the data block to be backed up is stored in the pack layer; In the current round of garbage collection process, all available class data fingerprints unique to expired snapshots are filtered out from available class data fingerprints, and all filtered available class data fingerprints are converted into forgettable class data fingerprints, and all backup processes occurring during the current round of garbage collection process are recorded; In the next round of garbage collection process, wait until all the recorded backup processes have ended, check whether any of the recorded backup processes has used the forgettable data fingerprint, and if a forgettable data fingerprint has been used by one or more recorded backup processes, convert the forgettable data fingerprint into an available data fingerprint; In the recovery process, the available data fingerprint and the forgettable data fingerprint are used to recover a snapshot of the data.

2. The data processing method according to claim 1, characterized in that: The method further comprises: When a data fingerprint of a data block to be backed up is identical to a data fingerprint of the data fingerprint library, it is determined that the data block to be backed up is a data block that has been backed up.

3. The data processing method according to claim 1, characterized in that: The method further comprises: When a forgettable data fingerprint has not been used by all the recorded backup processes, the forgettable data fingerprint is converted into a deletable data fingerprint; The deletable data fingerprint is deleted from the data fingerprint library, and the data block corresponding to the deletable data fingerprint is marked as deletable in the pack layer.

4. The data processing method according to claim 3, characterized in that: The deleting the deletable data fingerprint from the data fingerprint library includes: Move the deletable data fingerprint to the local recycle bin; After the first delay period, check whether the deletable data fingerprint has been used by any backup process that occurred within the first delay period; When the deletable data fingerprint has not been used by any backup process occurring within the first period of time, the deletable data fingerprint is deleted from the local recycle bin.

5. The data processing method according to claim 3, characterized in that: The step of marking the data block corresponding to the deletable data fingerprint as deletable in the pack layer includes: Record the data blocks corresponding to the deletable data fingerprints into the local recycle bin; If it is determined multiple times that the data block corresponding to the deletable data fingerprint has not been used by any backup process occurring within the first time period, the data block corresponding to the deletable data fingerprint is marked as deletable in a pack file, wherein the pack layer includes multiple pack files; When all data blocks in the pack file are not used by any backup process, the pack file is deleted from the local recycle bin; When the data block corresponding to the deletable data fingerprint has been used by one or more backup processes occurring within the first time period, the record of the data block corresponding to the deletable data fingerprint in the local recycle bin is revoked, and the deletable data fingerprint is converted into an available data fingerprint.

6. The data processing method according to claim 4, characterized in that: The method further comprises: Create a garbage backup repository for multiple user backup repositories belonging to the same region; Before performing garbage collection on the local recycle bin in each user backup repository, copy the garbage data in the local recycle bin in the user backup repository to the garbage backup repository; Delete the junk data in the local recycle bin in the user's backup library; The garbage data in the garbage backup library is deleted after a delay of a second time period, and the second time period is greater than the first time period.

7. The data processing method according to claim 6, characterized in that: The method further comprises: Before the second time period expires, when an abnormality occurs in a local recycle bin in a user backup repository, the garbage data in the local recycle bin in the user backup repository is restored using the garbage data in the garbage backup repository.

8. A data processing device, characterized in that: include: A first processing module is used to process each data block to be backed up in the backup process to obtain a data fingerprint of each data block to be backed up; A comparison module, used for comparing the data fingerprint of each data block to be backed up with each data fingerprint in the data fingerprint library; The second processing module is used for, when the data fingerprint of a data block to be backed up is different from each data fingerprint in the data fingerprint library, converting the data fingerprint of the data block to be backed up into a usable data fingerprint and storing it in the data fingerprint library, and storing the data block to be backed up in the pack layer; The third processing module is used to filter out all available class data fingerprints unique to expired snapshots from available class data fingerprints in the current round of garbage collection process, convert all filtered available class data fingerprints into forgettable class data fingerprints, and record all backup processes occurring during the current round of garbage collection process; The fourth processing module is used to wait for all the recorded backup processes to be completed in the next round of garbage collection process, check whether any of the recorded backup processes has used the forgettable data fingerprint, and if a forgettable data fingerprint has been used by one or more recorded backup processes, convert the forgettable data fingerprint into an available data fingerprint; The recovery module is used to recover a snapshot of the data using the usable data fingerprint and the forgettable data fingerprint.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images