A cloud hard disk backup and recovery system based on storage snapshots

Abstract

The present invention discloses a cloud hard disk backup and recovery system based on storage snapshots, which relates to the field of data storage technology and includes five modules: a cloud platform formulates snapshot, backup and recovery rules; a primary storage module stores data, generates snapshots and backs up data blocks before tampering to a secondary storage module; a reference strength analysis module calculates reference strength in combination with multi-dimensional data; a data block replication and diversion module monitors and regulates resources based on strength; and a secondary storage module receives or recovers data blocks when resources are normal. Each module collaborates to ensure backup and recovery efficiency and performance. The present invention quantifies the reference strength of data blocks, specifically replicates high-frequency and high-complexity data blocks, diverts high-frequency access objects, reduces the pressure of original data block backtracking, reduces CPU and IO consumption, and ensures cloud host performance. It also dynamically sets thresholds based on historical data, replicates when overloaded, deletes and monitors the replicated data blocks after the load drops, and balances resources and storage costs.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a cloud hard disk backup and recovery system based on storage snapshots. Background Art

[0002] Existing cloud hard drive backup solutions usually rely on the storage system to generate a complete cloud hard drive snapshot. The backup software then reads the snapshot in full on the cloud host, calculates the hash value, and compares it with the previous record to calculate the difference data. The difference data is then transferred to the backup storage, resulting in a large amount of CPU, IO and network bandwidth being consumed during the backup process, which in turn leads to performance degradation during backup. Summary of the Invention

[0003] Technical problems solved

[0004] In view of the shortcomings of the existing technology, the present invention provides a cloud hard disk backup and recovery system based on storage snapshots, which solves the problem of performance degradation during the cloud hard disk backup process.

[0005] Technical Solution

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cloud hard disk backup and recovery system based on storage snapshots, including the following specific modules: cloud platform: used to formulate rules for snapshot generation, data backup and data recovery; primary storage module: stores data to obtain data blocks, generates snapshots through the cloud platform and backs up the data blocks before tampering to the secondary storage module, and the snapshots are associated with the data blocks; reference intensity analysis module: obtains the number of references and reference depth of the snapshot in real time, calculates the number of references according to the time series, obtains the reference frequency, and performs standardization and comprehensive calculation based on the number of data blocks, the number of references, the reference depth and the reference frequency to obtain the reference intensity; data block replication and diversion module: used to monitor whether the reference intensity causes the CPU, IO and network bandwidth performance to decline and take measures; secondary storage module: when the CPU, IO and network bandwidth are normal, used to receive the backed-up data blocks or restore the backed-up data blocks to the primary storage module through the cloud platform.

[0007] Furthermore, the specific method for obtaining the citation strength is as follows: ;in, Indicates the citation strength, Indicates the number of data blocks, Indicates the breadth of citations, Indicates the The first data block The frequency of references to objects, Indicates the The first data block The reference depth of an object.

[0008] Furthermore, the specific steps for monitoring whether the citation intensity causes CPU, IO and network bandwidth performance degradation and taking measures are: setting a citation threshold, performing real-time comparison between the citation intensity and the citation threshold, continuing monitoring if the citation intensity is less than the citation threshold, and regulating through the data diversion algorithm if the citation intensity is greater than or equal to the citation threshold until the CPU, IO and network bandwidth are monitored to return to normal.

[0009] Furthermore, the specific steps of regulating through the data diversion algorithm are as follows: all referenced data blocks are weighted according to the reference strength to obtain the weight of each referenced data block, which is recorded as the first reference weight, and the first reference weights are sorted in order from small to large to obtain the largest first reference weight, and the data block corresponding to the largest first reference weight is recorded as A, and copied to obtain a copied data block, recorded as B, and B is backed up and stored in the secondary storage module, and then the objects referenced by A are weighted according to the reference frequency and reference depth to obtain the second reference weight, and the second reference weight is sorted in order from small to large, and the metadata of the object corresponding to the largest second reference weight is associated with B, and so on for other objects corresponding to the second reference weight, and the reference strength and the reference threshold are continued to be compared in real time. If the existence of A and B cannot make the reference strength less than the reference threshold, B continues to be copied, and so on.

[0010] Furthermore, the specific method for obtaining the second reference weight is as follows: ;in, represents the second citation weight, Indicates the frequency of citations, Indicates the reference depth.

[0011] Furthermore, the specific method of backing up the data block before being tampered with to the secondary storage module is as follows: this data block in the main storage module is stored at a new physical address through COW technology, and the new logical address is marked and associated with this data block, recorded as a new data block, and the data before being tampered with is backed up to the secondary storage module and associated with it through a snapshot.

[0012] Furthermore, the specific method of setting the reference threshold is as follows: obtaining historical normal data of CPU, IO and network bandwidth, performing comprehensive calculation on the historical normal data of CPU, IO and network bandwidth, and obtaining the reference threshold.

[0013] Furthermore, the specific steps of comprehensively calculating the historical normal data of CPU, IO and network bandwidth are: averaging the historical normal data of CPU, IO and network bandwidth to obtain normal balanced data, and then performing variance calculation based on the historical normal data of CPU, IO and network bandwidth and the normal balanced data.

[0014] Furthermore, the specific method of monitoring until the CPU, IO and network bandwidth return to normal is as follows: if the reference strength is less than the reference threshold, the cloud platform deletes the replicated data blocks in sequence according to the number of replicated data blocks in the secondary storage module until the CPU, IO and network bandwidth return to normal.

[0015] Furthermore, the cloud platform performs the following specific steps for deleting the duplicate data blocks in sequence according to the number of duplicate data blocks in the secondary storage module: each time a duplicate data block is deleted, a real-time comparison is continued between the reference strength and the reference threshold; if after deleting the duplicate data block, the reference strength is greater than or equal to the reference threshold, the duplicate data block is restored until the reference strength is less than the reference threshold; if after deleting the duplicate data block, the reference strength is still less than the reference threshold, a duplicate data block is deleted, and so on, until all the duplicate data blocks in the secondary storage module are deleted.

[0016] Beneficial effects

[0017] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0018] 1. By quantifying the reference intensity of data blocks, targeted replication is performed on data blocks with high frequency and high complexity. Reference paths of frequently accessed objects are directed to the replicated data blocks, reducing the multi-level backtracking pressure on the original data blocks. This reduces CPU metadata parsing and I / O operation consumption, ensuring stable cloud server performance during backup.

[0019] 2. Dynamically set reference thresholds based on historical resource data. When overloaded, replicated data blocks are used to divert pressure. When the load decreases, replicated data blocks are progressively deleted and monitored in real time. This improves system responsiveness while avoiding redundant storage and balancing resource utilization and storage costs.

[0020] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the present invention: a cloud hard disk backup and recovery system structure diagram based on storage snapshots.

[0022] Figure 2 The present invention is a line graph showing the impact of citation intensity on CPU resource utilization. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] It should be noted that, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include," "comprise," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0025] like Figure 1 As shown, an embodiment of the present invention provides a cloud hard disk backup and recovery system based on storage snapshots, which includes the following specific modules:

[0026] Cloud platform: Used to formulate rules for snapshot generation, data backup, and data recovery, automatically generating snapshots at set times. Snapshots are used to record the state of data blocks at a certain moment and to mark and associate the location and state of data blocks through metadata. Metadata includes the logical and physical addresses of data blocks, data block attributes, data block associations, data block status information, etc.

[0027] Primary storage module: The cloud platform stores data through the primary storage module to obtain data blocks. The data blocks are stored in physical addresses and associated with logical addresses. The data blocks are shared by the primary storage module and snapshots. The reference counting of the COW technology is used to count the number of shared objects of this data block. If the data block is tampered with, the primary storage module stores this data block at a new physical address and updates the metadata to mark and associate the new logical address with this data block, recording it as a new data block. The data block before the tampering is no longer shared by the primary storage module, but only by the snapshot associated with it. That is, the snapshot is associated with the metadata of this data block before it was tampered with, recorded as the original data block, and the cloud platform backs up the original data block to the secondary storage module. When a new data block is tampered with subsequently, the same process is repeated. This avoids the backup software from reading the full snapshot on the cloud platform, calculating the hash value, and calculating the difference data by comparing it with the original data block, and then transmitting the difference data to the backup storage, preventing the backup process from occupying a large amount of CPU or IO and network bandwidth, which reduces the performance of the backup.

[0028] Reference intensity analysis module: The number of snapshot references is directly obtained through the cloud platform's storage monitoring API or metadata interface to obtain the number of data block references, which is recorded as the reference breadth. The cloud platform's snapshot management API parses the dependency relationship to automatically build a hierarchical tree to obtain the referenced data block level, that is, the object that references the data block must traverse multiple levels to reference the data block, which is recorded as the reference depth. The cloud platform tracks the objects that reference this data block through metadata, and calculates the quotient based on the number of references to this object during system operation time to obtain the reference frequency of this object referencing this data block, which is recorded as the reference frequency. Since there are n data blocks in the main storage module, all these data blocks can be referenced. The reference breadth, reference depth, reference frequency and number of data blocks are standardized to eliminate the dimension, and a comprehensive calculation is performed to obtain the reference intensity, which reflects the intensity of the snapshot associated with the original data block or the new data block being referenced. The greater the intensity, the more CPU, IO and network bandwidth are consumed.

[0029] The specific method for obtaining citation strength is as follows:

[0030] ;

[0031] in, Indicates the reference intensity, which reflects whether a large amount of CPU, IO and network bandwidth is consumed. Indicates the number of data blocks, Indicates the breadth of citations, Indicates the The first data block The frequency of references to objects, Indicates the The first data block The reference depth of an object.

[0032] Table 1: The impact of a reference intensity on CPU resource utilization

[0033]

[0034] As shown in Table 1, in group 1, when the reference intensity is 93, the CPU resource utilization is 54%, in group 2, when the reference intensity is 126, the CPU resource utilization is 61%, and in group 3, when the reference intensity is 322, the CPU resource utilization is 74%. Figure 2 As shown in the figure, this indicates that when the complexity of the data block reference relationship processed by the system increases, the CPU resource utilization shows a nonlinear growth characteristic with the increase of reference intensity.

[0035] Data block replication and drainage module: used to monitor whether the reference intensity causes CPU, IO and network bandwidth performance degradation and take measures, set the reference threshold, and compare the reference intensity with the reference threshold in real time. If the reference intensity is less than the reference threshold, monitoring will continue. If the reference intensity is greater than or equal to the reference threshold, it means that more CPU, IO and network bandwidth are occupied. Therefore, the data drainage algorithm is used to regulate until the CPU, IO and network bandwidth are restored to normal. Only when the CPU, IO and network bandwidth return to normal can the data block be backed up or restored normally.

[0036] The specific steps for regulation through data diversion algorithm are as follows:

[0037] Assign weights to all referenced data blocks according to the reference strength to obtain the weight of each referenced data block, which is recorded as the first reference weight;

[0038] Sort the first reference weights in ascending order by bubble sorting to obtain the largest first reference weight;

[0039] The data block corresponding to the largest first reference weight, denoted as A, is copied to obtain a copied data block, denoted as B, and backed up and stored in the secondary storage module to avoid occupying the storage space of the primary storage module;

[0040] Then, weight the objects cited by A according to the citation frequency and citation depth to obtain the second citation weight;

[0041] Sort the second reference weights in ascending order through quick sorting, and associate the metadata of the object corresponding to the largest second reference weight with B, thereby reducing the association with A while maintaining the accuracy of the referenced data block. Since this object originally had the largest second reference weight when referencing A, after switching to referencing B, the computational complexity of the reference frequency and reference depth of this object when referencing A is reduced. This is analogous to other objects corresponding to the second reference weight. The presence of B also reduces the reference breadth and reference frequency of A, further reducing the reference breadth and reference frequency of objects when referencing A.

[0042] Continue to compare the citation strength with the citation threshold in real time. If the existence of A and B cannot make the citation strength less than the citation threshold, continue to copy B, and so on;

[0043] Except A, the same applies to other data blocks corresponding to the first reference weight.

[0044] The specific method for obtaining the second citation weight is as follows:

[0045] ;

[0046] in, represents the second citation weight, Indicates the frequency of citations, Indicates the reference depth.

[0047] The specific method for setting the reference threshold is as follows:

[0048] Obtain historical normal data of CPU, IO, and network bandwidth, perform comprehensive calculations on the historical normal data of CPU, IO, and network bandwidth, and obtain reference thresholds.

[0049] The specific steps of calculating the historical normal data of CPU, IO and network bandwidth by using the variance method are as follows:

[0050] The historical normal data of CPU, IO, and network bandwidth are averaged to obtain normal balanced data. This data is used as the standard for measuring the historical normal data of CPU, IO, and network bandwidth. The variance method is calculated based on the historical normal data of CPU, IO, and network bandwidth and the normal balanced data. The volatility of the historical normal data of CPU, IO, and network bandwidth on the normal balanced data reflects the range of the reference threshold.

[0051] The specific steps to monitor until the CPU, IO, and network bandwidth return to normal are as follows:

[0052] If the reference strength is less than the reference threshold, the cloud platform will delete the replicated data blocks in sequence according to the number of replicated data blocks in the secondary storage module. Each time a replicated data block is deleted, the reference strength will continue to be compared with the reference threshold in real time. If the reference strength is greater than or equal to the reference threshold after deleting this replicated data block, the replicated data block will be restored until the reference strength is less than the reference threshold. If the reference strength is still less than the reference threshold after deleting this replicated data block, another replicated data block will continue to be deleted, and so on, until all replicated data blocks are deleted in the secondary storage module to save storage space. Until the reference strength is still less than the reference threshold, it means that the CPU, IO and network bandwidth have returned to normal.

[0053] Secondary storage module: When the CPU, IO and network bandwidth are normal, it is used to receive the original data blocks backed up by the cloud platform. The cloud platform also references the original data blocks through the secondary storage module and restores them to the primary storage module. Only when the CPU, IO and network bandwidth are normal can the efficiency and integrity of data block backup or recovery be guaranteed.

[0054] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A cloud hard disk backup and recovery system based on storage snapshots, characterized by: Includes the following specific modules: Cloud platform: used to formulate rules for snapshot generation, data backup, and data recovery; Primary storage module: stores data, obtains data blocks, generates snapshots through the cloud platform, and backs up the data blocks before tampering to the secondary storage module. The snapshots are associated with the data blocks. Citation intensity analysis module: obtains the number of citations and citation depth of snapshots in real time, calculates the number of citations based on the time series to obtain the citation frequency, and performs normalization and comprehensive calculation based on the number of data blocks, number of citations, citation depth, and citation frequency to obtain the citation intensity; Data block replication and diversion module: used to monitor whether the reference intensity causes CPU, IO, and network bandwidth performance degradation and take appropriate measures; Secondary storage module: When the CPU, IO, and network bandwidth are normal, it is used to receive the backup data blocks or restore the backup data blocks to the primary storage module through the cloud platform; The specific method for obtaining the citation strength is as follows: ； in, Indicates the citation strength, Indicates the number of data blocks, The reference breadth is the number of times the data block is referenced. Indicates the The first data block The frequency of references to objects, Indicates the The first data block The reference depth of an object.

2. The cloud hard disk backup and recovery system based on storage snapshot according to claim 1, characterized in that: The specific steps for monitoring whether the citation intensity is abnormal and taking measures are as follows: Set a reference threshold and compare the reference strength with the reference threshold in real time. If the reference strength is less than the reference threshold, continue monitoring. If the reference strength is greater than or equal to the reference threshold, adjust and control it through the data diversion algorithm until the CPU, IO, and network bandwidth are monitored to return to normal.

3. The cloud hard disk backup and recovery system based on storage snapshot according to claim 2, characterized in that: The specific steps of regulating and controlling through the data diversion algorithm are as follows: All referenced data blocks are weighted according to the reference strength to obtain the weight of each referenced data block, which is recorded as the first reference weight. The first reference weights are sorted in ascending order to obtain the largest first reference weight. The data block corresponding to the largest first reference weight is recorded as A and copied to obtain a copied data block, which is recorded as B. The backup of B is stored in the secondary storage module. Then, objects referenced by A are weighted according to the reference frequency and reference depth to obtain the second reference weight. The second reference weights are sorted in ascending order, and the metadata of the object corresponding to the largest second reference weight is associated with B. The same is true for other objects corresponding to the second reference weight. Real-time comparison is continued through the reference strength and the reference threshold. If the existence of A and B cannot make the reference strength less than the reference threshold, B continues to be copied, and so on.

4. The cloud hard disk backup and recovery system based on storage snapshot according to claim 3, characterized in that: The specific method for obtaining the second reference weight is as follows: ； in, represents the second citation weight, Indicates the frequency of citations, Indicates the reference depth.

5. The cloud hard disk backup and recovery system based on storage snapshot according to claim 1, characterized in that: The specific method of backing up the data blocks before tampering to the secondary storage module is as follows: The data block in the primary storage module is stored at a new physical address through COW technology, and the new logical address is marked and associated with the data block, recorded as a new data block, and the data before tampering is backed up to the secondary storage module and associated with it through a snapshot.

6. The cloud hard disk backup and recovery system based on storage snapshot according to claim 2, characterized in that: The specific method of setting the reference threshold is as follows: Obtain historical normal data of CPU, IO, and network bandwidth, perform comprehensive calculations on the historical normal data of CPU, IO, and network bandwidth, and obtain reference thresholds.

7. The cloud hard disk backup and recovery system based on storage snapshot according to claim 6, characterized in that: The specific steps for comprehensively calculating the historical normal data of CPU, IO and network bandwidth are as follows: The historical normal data of CPU, IO and network bandwidth are averaged to obtain normal balanced data, and then the variance method is used to calculate the historical normal data of CPU, IO and network bandwidth and the normal balanced data.

8. The cloud hard disk backup and recovery system based on storage snapshot according to claim 2, characterized in that: The specific method for monitoring until the CPU, IO and network bandwidth return to normal is as follows: If the reference strength is less than the reference threshold, the cloud platform deletes the replicated data blocks in sequence according to the number of replicated data blocks in the secondary storage module until the CPU, IO, and network bandwidth return to normal.

9. The cloud hard disk backup and recovery system based on storage snapshot according to claim 8, characterized in that: The specific steps of the cloud platform deleting the duplicate data blocks in sequence according to the number of duplicate data blocks in the secondary storage module are as follows: Each time a replicated data block is deleted, a real-time comparison is continued between the reference strength and the reference threshold. If the reference strength is greater than or equal to the reference threshold after deleting the replicated data block, the replicated data block is restored until the reference strength is less than the reference threshold. If the reference strength is still less than the reference threshold after deleting the replicated data block, a replicated data block is deleted, and so on, until all replicated data blocks are deleted from the secondary storage module.

Images