A data disaster recovery method supporting multi-backup path switching

Abstract

The application discloses a kind of data disaster recovery methods of supporting multiple backup path switching, comprising, based on the sensitivity of input data and the generation of dynamic granularity data slice of storage medium characteristics, and generate the metadata label containing data slice identification, storage preference and recovery priority;Path quality atlas is generated for each data slice by constructing heterogeneous transmission path performance model through metadata label, and optimal transmission path is allocated, and path performance is monitored;Abnormality is detected and migrated slice by the standard deviation of cumulative transmission delay;Incremental integrity verification and Merkle proof of blockchain storage are used for parallel checking, and retransmission is preferentially checked failed slice;The application ensures the reliability and stability of data disaster recovery through the synergistic cooperation of dynamic data slicing, backup path optimization and recovery verification, to meet the real business needs.

Description

Technical Field

[0001] This invention relates to the field of data disaster recovery technology, and in particular to a data disaster recovery method that supports switching between multiple backup paths. Background Technology

[0002] With the rapid development of information technology, data has become a crucial asset for enterprises and organizations, and data disaster recovery technology has gradually become an important means of ensuring data security and business continuity. Traditional data disaster recovery methods typically use a single backup path for data transmission and storage, and the data granularity is fixed. They fail to consider differences in data sensitivity, storage media characteristics, and network path performance, making it difficult to meet the diverse needs of different business scenarios for data recovery speed and reliability in practical applications. In recent years, with the rise of cloud computing, distributed storage, and blockchain technologies, data disaster recovery technology has gradually developed towards refinement, intelligence, and efficiency.

[0003] However, existing data disaster recovery methods still have some shortcomings. First, traditional methods typically only consider the sensitivity differences of the data itself and the performance characteristics of the storage media, resulting in a lack of specificity in data sharding granularity and path selection strategies, making it difficult to achieve optimal performance in data backup and recovery. Second, traditional methods often ignore the real-time performance fluctuations of the transmission path and the response characteristics of the storage media during path selection, failing to establish an effective path quality assessment model, leading to a degree of blindness in path selection. Third, traditional methods lack integrity verification mechanisms during data recovery, especially failing to effectively utilize blockchain technology for parallel verification and rapid retransmission, resulting in a lengthy data recovery process that cannot meet the needs of rapid data recovery. Therefore, there is an urgent need for a data disaster recovery method that can consider data sensitivity, storage media characteristics, and transmission path performance, and supports switching between multiple backup paths.

[0004] CN118152186A discloses a data recovery method and apparatus in a distributed object storage system. It mainly utilizes an object recovery component to restore backup data from a tape library to a distributed file system, obtaining the complete content of the recovered object. By scanning the entire content of the recovered object, the metadata of each object is parsed and written to a distributed database. Then, when an object retrieval request is received from a client, the corresponding object is retrieved from the distributed file system based on its metadata. Although metadata is mentioned, this scheme does not further consider metadata, resulting in an inability to account for the sensitivity differences of the data itself and the performance characteristics of the storage medium.

[0005] CN117149526A discloses a data recovery method, system, and device suitable for MPP databases. It primarily utilizes a client to initiate a data backup request, import the information to be backed up, connect to the MPP database, read metadata information, and obtain the MPP database backup table, metadata information, and the location of the data files stored in HDFS. The data files are then backed up from HDFS to object storage. Next, the corresponding backup table is selected, and the path of the backup files on object storage is obtained through the recorded backup information. The specified path on object storage is retrieved by calling the object storage API, and the files are restored to the MPP database. This solution mainly backs up to the object storage file system rather than the storage medium. While it considers the backup path, it lacks consideration for the storage medium, making it difficult to meet the data recovery speed requirements of different business scenarios. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0007] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a data disaster recovery method that supports multi-backup path switching to solve the problems mentioned in the background art.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: a data disaster recovery method supporting multi-backup path switching, comprising:

[0009] The system acquires input data, generates dynamic-granular data fragments based on the sensitivity of the input data and the characteristics of the storage medium, and generates metadata tags for each data fragment.

[0010] The performance model of the heterogeneous transmission path is constructed using the metadata tags, and a path quality map is generated.

[0011] Based on the metadata tags and path quality map, an optimal transmission path is assigned to each data fragment, and the performance of the assigned optimal transmission path is monitored.

[0012] Based on the monitoring results, incremental integrity verification is performed on the recovered data fragments, and Merkle proofs generated and stored in the blockchain using metadata tags are used for parallel verification. Data fragments that failed verification are retransmitted, thereby achieving efficient backup and priority recovery of data fragments.

[0013] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, the method includes: acquiring input data; generating dynamically granular data fragments based on the sensitivity and storage medium characteristics of the input data; and generating metadata tags for each data fragment, including:

[0014] A sensitivity score is generated based on the entropy density, access pattern ratio, and structural correlation of the input data.

[0015] The compatibility between data fragments and storage media is calculated based on the sequential read bandwidth, random write latency, and maximum IOPS of the storage medium.

[0016] The data granularity is determined based on the sensitivity score and the fit, and metadata tags containing data shard identifiers, storage preferences, and recovery priorities are generated.

[0017] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, it further includes:

[0018] The data shard identifier in the metadata tag is generated using a hash value. The storage preference in the metadata tag is determined by the storage medium type and the data shard I / O mode. The recovery priority in the metadata tag is calculated based on the sensitivity score and the business SLA coefficient.

[0019] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, the method includes: constructing a performance model of heterogeneous transmission paths and generating a path quality map through the metadata tags, including:

[0020] Based on the storage preferences in the metadata tags, multiple transmission paths from the source node to the backup storage node are defined, and a performance model for heterogeneous transmission paths is constructed.

[0021] Based on the performance model of the heterogeneous transmission path, the response time of the path medium characteristics is predicted according to the storage medium characteristics and real-time queue requests in the storage preference. At the same time, considering the transmission success rate and latency fluctuation of the transmission path, the network stability is predicted.

[0022] Based on the response time and network stability of the obtained path medium characteristics, a path quality map is constructed.

[0023] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, the method includes: allocating an optimal transmission path for each data fragment based on the metadata tags and path quality map, including:

[0024] Based on storage preferences and recovery priorities in metadata tags and the resulting path quality map, a multi-dimensional decision matrix is ​​constructed that includes media matching degree, network stability, and recovery priority.

[0025] Based on the multidimensional decision matrix, an optimal transmission path is assigned to each data segment, and the assignment of the optimal transmission path to each data segment is treated as a constrained optimization problem. The constrained optimization problem is then solved using an improved Hungarian algorithm.

[0026] As a preferred embodiment of the data disaster recovery method supporting multiple backup path switching described in this invention, the media matching degree is the degree of matching between the data fragmentation I / O mode and the characteristics of the path media.

[0027] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, the improved Hungarian algorithm includes:

[0028] The medium matching degree, network stability, and recovery priority in the multidimensional decision matrix are transformed into a single weight matrix by weighted summation, and constraints on transmission path load, recovery time, and resources are established.

[0029] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, monitoring the performance of the allocated transmission path includes:

[0030] The performance of the allocated transmission path is determined by the standardized deviation of the accumulated transmission delay. When an abnormality is detected in the performance of the allocated transmission path, the data fragments are migrated to a new transmission path based on the path quality map.

[0031] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, the method includes: performing incremental integrity verification on the recovered data fragments based on monitoring results, including:

[0032] The data fragments in the backup storage are located based on the data fragment identifier in the metadata tag. The contents of the data fragments are read, a Merkle proof is generated for each data fragment whose data contents are read, and the Merkle root hash is written into the lightweight blockchain.

[0033] As a preferred embodiment of the data disaster recovery method supporting multi-backup path switching described in this invention, the method includes: using Merkle proofs generated and stored in the blockchain using metadata tags for parallel verification, and retransmitting data fragments that failed verification, including:

[0034] The thread pool is used to verify each data fragment of the read data content in parallel, and the data fragment identifier is matched with the generated Merkle proof.

[0035] If there are data fragments that fail to be validated in parallel using a thread pool, retransmission will be triggered based on the recovery priority in the metadata tag.

[0036] Compared with existing technologies, the beneficial effects of the invention are:

[0037] 1. By generating dynamically granular data fragments based on input data sensitivity and storage media characteristics, and combining storage preferences and recovery priorities in metadata tags, this invention achieves efficient adaptation between data fragments and storage media. Compared to traditional methods, this invention can dynamically adjust the granularity of data fragments according to data characteristics and business needs, thereby improving data backup efficiency.

[0038] 2. By using an improved Hungarian algorithm for multi-objective optimization, the optimal transmission path is assigned to each data fragment. At the same time, the performance of the transmission path is monitored in real time, and fragments are dynamically migrated to new transmission paths when anomalies are detected. This effectively copes with network fluctuations and storage media performance degradation, ensuring the stability and reliability of data transmission and overcoming the limitation of traditional path selection that easily leads to data backup failure.

[0039] 3. An incremental integrity verification mechanism is adopted, which performs parallel verification only on the recovered fragments. The data fragment identifier in the metadata tag is used to match the Merkle proof stored in the lightweight blockchain. Compared with traditional verification methods, this invention can quickly verify and recover data. At the same time, the storage of the lightweight blockchain ensures the immutability and traceability of the verification results. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0041] Figure 1 This is a flowchart illustrating the overall process of a data disaster recovery method supporting multiple backup path switching according to an embodiment of the present invention. Detailed Implementation

[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0044] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0045] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0046] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0047] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0048] Example 1

[0049] Reference Figure 1 This is the first embodiment of the present invention, which provides a data disaster recovery method supporting multiple backup path switching, including:

[0050] S1. Obtain input data, generate dynamic granular data fragments based on the sensitivity of the input data and the characteristics of the storage medium, and generate metadata tags for each data fragment;

[0051] Specifically, acquiring input data refers to the raw data that needs to be backed up and disaster recovered, including database data (such as relational database tables, NoSQL documents, key-value stores), file system data (such as documents, log files, configuration files), multimedia data (such as images, videos), and real-time data (such as log streams, sensor data, transaction records).

[0052] Furthermore, a sensitivity score is generated based on the entropy density, access pattern ratio, and structural correlation of the input data;

[0053] Specifically, based on the Shannon entropy formula, the randomness of the data is calculated, and the entropy density E of the data is obtained:

[0054] E=―∑p(a i )logP(a I )

[0055] Wherein, P(a i ) represents the byte value a in the data. I The probability of occurrence of (range 0 to 255);

[0056] It should be noted that the entropy density of data reflects the randomness and information content of the data, and the unit is bits per byte. The higher the value, the more sensitive information the data contains (such as encrypted data). For example, encrypted databases often have a higher entropy density than log files.

[0057] Specifically, the access pattern ratio refers to the proportion of data accessed during read / write operations. It is dimensionless and ranges from [0,1]. A higher value indicates more frequent data access, requiring priority backup. For example, sequential reads and random writes. Frequently accessed data is usually considered sensitive information (such as usernames and passwords) and requires priority backup and recovery. In actual processing, the access pattern ratio is calculated using a sliding window to determine the proportion of data accessed during read / write operations.

[0058]

[0059] Where wn represents the number of data write operations; rn represents the number of data read operations;

[0060] Specifically, structural correlation refers to the organization or dependency relationship between data. It is dimensionless and ranges from [0,1]. The higher the value, the stronger the data correlation, and the more necessary it is to perform a full backup. Examples include foreign keys in database tables and directory structures in file systems. In particular, data tables with strong correlations (master tables and slave tables) need to be backed up as a whole.

[0061] It should be explained that, in practical applications, the organization and dependencies between data are very complex. It is necessary to use existing graph theory methods to identify which data tables have strong relationships. Graph theory methods can abstract tables into nodes and foreign key relationships into edges. Edge weights are assigned based on the frequency of fields in the table or the amount of data interaction between tables to form a directed graph. This directed graph is then processed to identify data tables with strong relationships. Finally, by calculating the proportion of data tables with strong relationships among all data tables with relationships (including strong and weak relationships), the quantified structural relationship can be obtained.

[0062] Specifically, to unify the units of measurement and balance the influence of various factors, the sensitivity score is calculated using a weighted summation method. The resulting sensitivity score formula is as follows:

[0063] S=ω1·norm(E)+ω2·A+ω3·R scd

[0064] Where S represents the sensitivity score, a higher value indicates higher data sensitivity, requiring finer-grained sharding and higher recovery priority, with a value range of [0,1]. A represents the access pattern ratio, and R... scd ω1 represents the structural relevance; v1, ω2, and ω3 represent the sensitivity score, access pattern ratio, and structural relevance weight coefficients, respectively; norm represents the normalization operation, and the normalization operation in this invention adopts Min-Max normalization.

[0065] For example, for an encrypted database, assume norm(E) = 0.9375, A = 0.8, R scd =0.9, with weighting coefficients of 0.4, 0.3, and 0.3 being classic values ​​(e.g., obtained through business scenario analysis or machine learning optimization), resulting in:

[0066] S=0.4·0.9375+0.3·0.8+0.3·0.9=0.885

[0067] Conclusion: This sensitivity score indicates that the data is highly sensitive, requiring more granular sharding and a higher recovery priority;

[0068] It should be noted that the sensitivity score demonstrates the sensitivity of the data and also determines the granularity of subsequent data sharding and the priority of data recovery.

[0069] Furthermore, based on the sequential read bandwidth, random write latency, and maximum IOPS of the storage medium, the compatibility between data fragments and the storage medium is calculated;

[0070] It should be explained that storage media characteristics refer to the performance parameters of storage media (such as NVMe SSD, SATA HDD), which include the sequential read bandwidth, random write latency and maximum IOPS of the storage media;

[0071] Specifically, the sequential read bandwidth of a storage medium refers to the maximum data transfer rate (MB / s or GB / s) when the storage medium is continuously reading data. This sequential read bandwidth is obtained from the storage manufacturer's specification sheet. Its main function is to quickly transfer data in a sequential read bandwidth manner when backing up large files (such as database snapshots).

[0072] Specifically, random write latency refers to the average time (μs or ms) required for a storage medium to complete a random write operation, and is mainly used for performance constraints of data fragmentation transmission paths;

[0073] Specifically, maximum IOPS (Input / Output Operations Per Second) refers to the maximum number of read and write operations that the storage medium can process per second. Its main function is to allow multiple data fragments to be transmitted in parallel during backup processing. In addition, it can also verify the integrity of multiple data fragments during data recovery.

[0074] Specifically, the bandwidth R is read sequentially according to the aforementioned storage medium. seq Random write latency W rand And the maximum IOPS, define the storage medium characteristic vector M:

[0075] M = (R seq W rand IOPS max )

[0076] Among them, IOPS max Represented as maximum IOPS;

[0077] Specifically, the fit between data fragments and storage media is calculated using storage media characteristic vectors:

[0078]

[0079] Where F represents the compatibility between the data fragment and the storage medium, indicating the degree of matching between the storage medium and the data fragment; a higher value indicates a more suitable allocation. r ·S·norm(R seq This reflects the data sensitivity's need for high-bandwidth storage, suitable for large file backups. s This is represented as the weighting coefficient of that item; This reflects the storage medium's support for random access, making it suitable for parallel processing of small data fragments. ωm This is represented as the weighting coefficient for that item;

[0080] Furthermore, the granularity of data sharding is determined based on sensitivity scores and suitability, and metadata tags containing data shard identifiers, storage preferences, and recovery priorities are generated.

[0081] Specifically, the defined data granularity N is represented as:

[0082]

[0083] Among them, D total This represents the total data capacity, and scale is the scaling factor used to control the number of shards, with a default value of 100.

[0084] It should be noted that when implementing multi-path backup transmission, if the data fragment size exceeds the path's maximum transmission unit (MTU), the network layer will fragment the data packets, which will lead to CPU and network load, as well as the risk of packet loss. Therefore, it is necessary to impose upper and lower limits on the determination of the data fragment granularity.

[0085] Specifically, the upper limit constraint rule is: the size of a single data shard shall not exceed 90% of the current minimum backup path MTU; the lower limit rule is: each data shard shall contain at least 3 complete data units.

[0086] It should be noted that because the MTU includes protocol headers (such as IP header, TCP header, and application layer header), by setting 90% as the upper limit of the data fragment size, space can be reserved for protocol headers and potential additional encapsulation (such as VPN and tunneling protocols). For example, if the MTU is 1500 bytes, the protocol header occupies 40 bytes, and 90% is 1350 bytes, it can ensure that the actual payload of the data fragments fits the protocol header.

[0087] Furthermore, the data shard identifier in the metadata tag is generated using a hash value, the storage preference is determined by the storage medium type and the data shard I / O mode, and the recovery priority in the metadata tag is calculated based on the sensitivity score and the business SLA coefficient.

[0088] Specifically, a hash value for each data fragment is generated using the SHA-256 hash algorithm to identify the data fragment.

[0089] Specifically, the storage media types in storage preferences include common electrical storage devices (such as USB flash drives and SD cards), magnetic storage devices (such as hard drives and magnetic tapes), and optical storage devices (such as optical discs).

[0090] It should be noted that, for the data fragmentation I / O mode, the solution of this invention only considers the cases of sequential read and random write;

[0091] Furthermore, the data fragmentation I / O mode is obtained by calculating the locality index of the access address and the time weight of the access, where the locality index L of the access address is... s The calculation formula is as follows:

[0092]

[0093] Among them, addr l Let represent the address accessed for the l-th time, n be the number of accesses, and max(addr) represent the maximum access address.

[0094] It should be noted that if there is no previously accessed address (previous address), it is ignored. Usually, the starting address does not have a previously accessed address (previous address).

[0095] Specifically, the locality index represents the degree of concentration of accessed addresses, with low values ​​indicating high locality (random access, small address changes) and high values ​​indicating low locality (sequential access, large address jumps).

[0096] Furthermore, the time weighting of access L t The calculation formula is as follows:

[0097]

[0098] Where λ represents the access time decay factor, hit(l) indicates whether the l-th access is effective (takes a value of 0 or 1), and t current Indicates the current access time, t l This is represented as the time of the l-th visit;

[0099] Specifically, the time weighting of access reflects the frequency of data access;

[0100] It should be noted that by using the access time weight and the locality index of the access address mentioned above, we can obtain the number of times the current storage medium type accesses the address and the access time, thereby determining the storage preference during backup.

[0101] For example, suppose we analyze the database access logs and collect the 10 most recent access records (in reality, there may be hundreds). The access logs include the access address and access time; the access records are shown in Table 1 below:

[0102] Table 1

[0103] Access sequence number (l) <![CDATA[Access address (addr l )]]> <![CDATA[Access time (t l )]]> Is it effective (hit(l))? 1 100 0 1 2 102 1 1 3 101 2 1 4 200 3 1 5 201 4 1 6 300 5 0 7 103 6 1 8 104 7 1 9 105 8 1 10 106 9 1

[0104] Parameter settings: max(addr) = 10,000KB (assuming a total database size of 10GB), n = 10, λ = 0.9, t current=10, data unit is 1KB (per record);

[0105] According to the locality index L of the access address s Calculate the address difference for each access:

[0106] Access sequence number 1: No preceding address, ignore;

[0107] Access number 2: |addr2―addr 2―1 |=|102―100|=2KB

[0108] Access number 3: |addr3―addr 3―1 |=|101―102|=1KB

[0109] Access number 4: |addr4―addr 4―1 |=|200―101|=99KB

[0110] Access number 5: |addr5―addr 5―1 |=|201―200|=1KB

[0111] Access number 6: |addr6―addr 6―1 |=|300―201|=99KB

[0112] Access number 7: |addr7―addr 7―1 |=|103―300|=197KB

[0113] Access number 8: |addr8―addr 8―1 |=|104―103|=1KB

[0114] Access number 9: |addr9―addr 9―1 |=|105―104|=1KB

[0115] Access sequence number 10: |addr 10 ―addr 10―1 |=|106―105|=1KB

[0116] After removing access number 1, normalizing the difference, and dividing by max(addr), we get:

[0117] Access sequence number 2: 2 / 10,000 = 0.0002;

[0118] Access sequence number 3: 1 / 10,000 = 0.0001;

[0119] Access sequence number 4: 99 / 10,000 = 0.0099;

[0120] Access sequence number 5: 1 / 10,000 = 0.0001;

[0121] Access sequence number 6: 99 / 10,000 = 0.0099;

[0122] Access sequence number 7: 197 / 10,000 = 0.0197;

[0123] Access sequence number 8: 1 / 10,000 = 0.0001;

[0124] Access sequence number 9: 1 / 10,000 = 0.0001;

[0125] Access sequence number 10: 1 / 10,000 = 0.0001;

[0126] Calculate the average value after summing. get:

[0127] 0.0002 + 0.0001 + 0.0099 + 0.0001 + 0.0099 + 0.0197 + 0.0001 + 0.0001 + 0.0001 = 0.0402

[0128] 0.0402 / 9 ≈ 0.0045 (rounded to 4 decimal places)

[0129] As can be seen, a low locality index (0.0045) indicates that the access address changes little and the access pattern is random access, which is suitable for high IOPS storage media.

[0130] Based on the time weight of the access L t Calculate the weight of each visit (rounded to 4 decimal places):

[0131] Access sequence number 1: t1 = 0, λ = 1, 0.9 10―0 ×1=0.9 10 ≈0.3487

[0132] Access sequence number 2: t1 = 1, λ = 1, 0.9 10―1 ×1=0.9 9 ≈0.3874

[0133] Access sequence number 3: t1 = 2, λ = 1, 0.9 10―2 ×1=0.9 8 ≈0.4305

[0134] Access sequence number 4: t1 = 3, λ = 1, 0.9 10―3 ×1=0.9 7 ≈0.4783

[0135] Access sequence number 5: t1 = 4, λ = 1, 0.910―4 ×1=0.9 6 ≈0.5314

[0136] Access sequence number 6: t1 = 5, λ = 0, 0.9 10―5 ×0=0

[0137] Access sequence number 7: t1 = 6, λ = 1, 0.9 10―6 ×1=0.9 4 =0.6561

[0138] Access sequence number 8: t1 = 7, λ = 1, 0.9 10―7 ×1=0.9 3 =0.7290

[0139] Access sequence number 9: t1 = 8, λ = 1, 0.9 10―8 ×1=0.9 2 =0.8100

[0140] Access sequence number 10: t1 = 9, λ = 1, 0.9 10―9 ×1=0.9 1 =0.9000

[0141] Summation yields

[0142] 0.3487+0.3874+0.4305+0.4783+0.5314+0.6561+0.7290+0.8100+0.9000≈5.2714 As can be seen, because the time weight of access accounts for more than half (the upper limit of time weight is 10), it indicates that access is frequent and recent access is the main access method (the weight increases from 0.3487 to 0.9), so a storage medium with high-frequency access is needed;

[0143] Based on the locality index of the access address and the time weight of the access, the storage preference can be obtained, that is, to select storage media with high IOPS, suitable for random read and write and with high frequency of access (such as Alibaba Cloud ESSD, NVMe SSD).

[0144] Specifically, the business SLA coefficient refers to the contract between an enterprise and a service provider or internal IT system, which specifies the performance indicators of the service (such as recovery time targets, recovery point targets, availability), and is usually specified by the system administrator or the business party.

[0145] Specifically, the recovery priority P is determined based on the business's SLA coefficient and sensitivity score:

[0146] P = S × SLA coeff

[0147] Among them, SLAcoeff Represented as the business SLA coefficient;

[0148] S2. Construct a performance model for heterogeneous transmission paths and generate path quality maps by using metadata tags;

[0149] Furthermore, based on storage preferences in metadata tags, multiple transmission paths are defined from the source node to the backup storage node, and a performance model for heterogeneous transmission paths is constructed.

[0150] Each transmission path connects to different storage media and has different network characteristics;

[0151] It should be explained that, in the present invention, the transmission path refers to the data transmission channel from the source node to the backup storage node. These paths are heterogeneous (i.e. have different network characteristics) and are used to transmit data fragments to the backup storage node.

[0152] Furthermore, by using the performance model of heterogeneous transmission paths, based on the storage medium characteristics and real-time queue requests in storage preferences, the response time of the path medium characteristics is predicted. At the same time, considering the transmission success rate and latency fluctuations of the transmission path, network stability is predicted.

[0153] Specifically, the response time of path media characteristics refers to the response time of the storage media at the backup storage node at the end of the path; path media characteristics refer to the performance of the storage media involved in the transmission path.

[0154] Specifically, real-time queue requests refer to the current I / O queue depth and request type (read / write, sequential / random); where the current I / O queue depth refers to the number of I / O requests that the backup storage node is currently processing.

[0155] Specifically, the response time T of the predicted path medium characteristics med :

[0156]

[0157] Where β is the bandwidth attenuation coefficient (dimensionless, default value is 0.5, adjusted according to the storage medium type, such as 0.3 for NVMe SSD and 0.7 for HDD); N req ql represents the number of data fragment requests, i.e., the number of fragments to be transmitted; ql represents the number of pending requests on the storage medium, i.e., the current I / O queue depth, which is obtained in real time through the storage system API (such as Linux / proc / diskstats); This represents the transmission delay caused by bandwidth limitations; This indicates the processing latency caused by queue depth and random write latency;

[0158] For example, assuming β = 0.5, N req =100, R seq =500MB / s, ql=50, W rand =0.1, IOPS max =100000, resulting in:

[0159]

[0160] Conclusion: A response time of 0.10005 ms indicates the efficiency of the storage medium in handling the current load;

[0161] It's important to explain that bandwidth increases the throughput of storage media, thereby enhancing its ability to respond to requests, and IOPS determines the response speed of storage media under high load. Therefore, both factors influence the maximum response capability of storage media. Furthermore, high latency means a slower response from storage media to requests, while low latency means a faster response. For small requests (such as 4KB), low latency is more crucial because the overhead of each operation is significant. For large requests (such as 1MB), bandwidth is more important, and latency has a smaller impact.

[0162] Specifically, predicting network stability S index :

[0163]

[0164] Where success_rate represents the weighted average success rate. If 9 out of 10 transmissions are successful, then success_rate = 0.9. The latency fluctuation coefficient is represented by μ, which reflects network jitter. The larger the fluctuation coefficient, the worse the stability. μ (latency) represents the mean latency, and σ (latency) represents the standard deviation of latency.

[0165] Specifically, a path quality map is constructed based on the response time and network stability of the obtained path medium characteristics;

[0166] It should be noted that the path quality map includes path (path_id), storage media type (media_type), response time of path media characteristics (T_media), and network stability (Stability_index); for example, it can be assigned by code as: [{"path_id":1,"media_type":"NVMe SSD","T_media":1.02,"Stability_index":0.99}];

[0167] S3. Based on metadata tags and path quality maps, allocate the optimal transmission path for each data fragment and monitor the performance of allocating the optimal transmission path;

[0168] Furthermore, based on storage preferences and recovery priorities in the metadata tags and the resulting path quality map, a multi-dimensional decision matrix including media matching degree, network stability, and recovery priority is constructed.

[0169] Specifically, media compatibility M atc h med,ij The degree of matching between data fragmentation I / O patterns and path medium characteristics is calculated using cosine similarity as follows:

[0170]

[0171] in, Represented as a data fragmentation I / O mode vector, g1 is the sequential read ratio (based on the access mode ratio A), and g2 is the random write ratio (1-A). Represented as a path medium characteristic vector, cos(θ) represents the cosine similarity; ||·|| represents the norm of the vector, that is, the length or size of the vector;

[0172] For example, assuming the data fragmentation access pattern ratio A = 0.8, then Path get:

[0173]

[0174] (Retain four decimal places)

[0175] (Retain four decimal places)

[0176] (Retain four decimal places)

[0177] Conclusion: A score of 0.2907 indicates a low degree of matching between data fragmentation and path media, requiring further optimization of path allocation;

[0178] Specifically, each item in the multidimensional decision matrix represents the overall score of a data slice i being assigned to a path j;

[0179] Furthermore, based on the multidimensional decision matrix, an optimal transmission path is assigned to each data segment, and the assignment of the optimal transmission path to each data segment is treated as a constrained optimization problem. The constrained optimization problem is then solved using an improved Hungarian algorithm.

[0180] It should be explained that the Hungarian algorithm is a classic bipartite graph matching algorithm, mainly used to solve assignment problems. In the application scenario of this invention, its principle is to find the maximum weight and match it, so that each data piece can be assigned a unique path and the total weight (score) is maximized. The traditional Hungarian algorithm assumes one-to-one matching, cannot directly handle multi-objective optimization and does not support complex constraints. In order to handle the multi-objective optimization problem in the solution of this invention, it needs to be improved.

[0181] Specifically, the improved Hungarian algorithm is as follows:

[0182] The medium matching degree, network stability, and recovery priority in the multidimensional decision matrix are transformed into a single weight matrix by weighted summation, and constraints on transmission path load, recovery time, and resources are established.

[0183] It should be noted that the purpose of converting to a single weight matrix is ​​to transform a multi-objective optimization problem into a single-objective optimization problem;

[0184] Specifically, a single weight matrix M ij Represented as:

[0185]

[0186] in, The weight ω represents the medium matching degree. P The weights, ω, represent network stability. S The weights are represented as recovery priorities.

[0187] Specifically, this can be transformed into a single-objective optimization problem as follows:

[0188]

[0189] Specifically, establish transmission path load constraints, meaning each path can carry a maximum of a limited number of high-priority fragments:

[0190]

[0191] Among them, y ij This indicates that the data shard is a high-priority shard and is assigned to path j; It can be represented as arbitrary;

[0192] Specifically, establish a recovery time constraint, namely the total recovery time T. total Meeting the recovery time objective (RTO):

[0193]

[0194] Where, x ij This is represented as data sharding;

[0195] Specifically, establish resource constraints, namely, bandwidth and IOPS must not be exceeded:

[0196]

[0197] It should be noted that by using the improved Hungarian algorithm to assign the optimal transmission path to each data fragment, the stability of the backup data during transmission can be ensured.

[0198] It should be explained that if the performance of the allocated transmission path is abnormal, it may cause data fragmentation transmission failure or delay, thereby affecting the integrity of the backup and increasing the risk of data loss during disaster recovery. Therefore, it is necessary to make a judgment on this.

[0199] Furthermore, the performance of the allocated transmission path is judged by the standardized deviation of the accumulated transmission delay. When the performance of the allocated transmission path is detected to be abnormal, the data fragments are migrated to the new transmission path based on the path quality map.

[0200] Specifically, transmission delay data is collected once per second and stored in a time window. The cumulative transmission delay is obtained by summing all delays within the time window. Then, the standardized deviation of the cumulative transmission delay is calculated, and an anomaly threshold is set based on expert experience (e.g., if the standardized deviation of the cumulative transmission delay > 2, i.e., the performance fluctuation exceeds twice the mean). If the anomaly threshold is exceeded, the transmission path is marked as an anomaly, the data fragment transmission on the anomaly path is paused, the current anomaly path (path_id) is recorded, and the anomaly path (path_id) is removed based on the path quality map. The metadata label is updated. The path with the highest score in the multidimensional decision matrix is ​​recalculated as the new path, and the data fragment is migrated to the new path for transmission.

[0201] S4. Based on the monitoring results, perform incremental integrity verification on the recovered data fragments, and use the Merkle proof generated by the metadata tags and stored in the blockchain for parallel verification. Retransmit the data fragments that failed the verification, thereby achieving efficient backup and priority recovery of data fragments.

[0202] Furthermore, based on the data shard identifier in the updated metadata tag, the data shard in the backup storage is located, the data shard content is read, a Merkle proof is generated for each data shard whose data content is read, and the Merkle root hash is written into the lightweight blockchain (private chain);

[0203] It should be explained that Merkle proof is a data structure based on a Merkle tree (a type of binary hash tree). In the present invention, it is mainly used to prove whether a certain data shard belongs to the data shard set and to verify its integrity and consistency.

[0204] Specifically, the updated data shard identifier is used as the key to query the backup storage, retrieve the content of the data shard, and verify whether the data shard exists. If it does not exist, it is marked as a failure; if it exists, a Merkle proof is generated. A hash value is calculated for each data shard content, which is consistent with the initial hash value generated by the data shard identifier generation method (i.e., using SHA-256 hashing) to obtain the shard hash value. The shard hash value is used as a leaf node, and the hash value is calculated by pairing leaf nodes with each other layer by layer until the root node is reached to obtain the Merkle root hash. The Merkle root hash is then written into the lightweight blockchain.

[0205] It should be noted that the immutability of blockchain can ensure the trustworthiness of the Merkle root hash for data shard verification, thus preventing data from being maliciously modified. In addition, blockchain can also provide timestamps and historical records, which facilitates data traceability.

[0206] Furthermore, a thread pool is used to verify each data shard of the read data content in parallel, matching the data shard identifier with the generated Merkle proof;

[0207] Specifically, a fixed-size thread pool (e.g., 8 threads, based on the number of CPU cores) is created, and a thread-safe queue (e.g., Python's concurrent.futures.ThreadPoolExecutor) is used to allocate one thread to each data shard and rebuild the Merkle root hash. When the rebuilt Merkle root hash matches the Merkle root hash written to the blockchain, the verification is successful; otherwise, the verification fails.

[0208] Furthermore, if there are data fragments that fail to be validated in parallel using a thread pool, retransmission is triggered based on the recovery priority in the metadata tag.

[0209] Specifically, the data fragments that failed verification and their metadata tags are sorted according to recovery priority, and steps S2 to S4 are re-executed (during this period, the data fragment status in the backup storage is updated) until the retry limit (e.g., 3 times) is reached.

[0210] It should be noted that by retransmitting only the data fragments that failed the verification, other data fragments are not affected, thus avoiding the duplicate transmission of data fragments and meeting the requirements for rapid data disaster recovery.

[0211] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0212] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0216] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data disaster recovery method supporting multiple backup path switching, characterized in that, include: The system acquires input data, generates dynamic-granular data fragments based on the sensitivity of the input data and the characteristics of the storage medium, and generates metadata tags for each data fragment. The process involves acquiring input data, generating dynamically granular data fragments based on the sensitivity of the input data and the characteristics of the storage medium, and generating metadata tags for each data fragment, including: A sensitivity score is generated based on the entropy density, access pattern ratio, and structural correlation of the input data. The compatibility between data fragments and storage media is calculated based on the sequential read bandwidth, random write latency, and maximum IOPS of the storage medium. The data granularity is determined based on the sensitivity score and the fit, and metadata tags containing data shard identifiers, storage preferences, and recovery priorities are generated. Also includes: The data shard identifier in the metadata tag is generated using a hash value; the storage preference in the metadata tag is determined by the storage medium type and data shard I / O mode; and the recovery priority in the metadata tag is calculated based on sensitivity score and business SLA coefficient. The performance model of the heterogeneous transmission path is constructed using the metadata tags, and a path quality map is generated. Based on the metadata tags and path quality map, an optimal transmission path is assigned to each data fragment, and the performance of the assigned optimal transmission path is monitored. Based on the aforementioned metadata tags and path quality map, an optimal transmission path is assigned to each data fragment, including: Based on storage preferences and recovery priorities in metadata tags and the resulting path quality map, a multi-dimensional decision matrix is ​​constructed that includes media matching degree, network stability, and recovery priority. Based on the multidimensional decision matrix, an optimal transmission path is assigned to each data segment, and the assignment of the optimal transmission path to each data segment is taken as a constrained optimization problem. The constrained optimization problem is solved by an improved Hungarian algorithm. The improved Hungarian algorithm includes: The medium matching degree, network stability, and recovery priority in the multidimensional decision matrix are transformed into a single weight matrix by weighted summation, and constraints on transmission path load, recovery time, and resources are established. Based on the monitoring results, incremental integrity verification is performed on the recovered data fragments, and Merkle proofs generated and stored in the blockchain using metadata tags are used for parallel verification. Data fragments that failed verification are retransmitted, thereby achieving efficient backup and priority recovery of data fragments.

2. The data disaster recovery method supporting multiple backup path switching as described in claim 1, characterized in that, Using the aforementioned metadata tags, a performance model for heterogeneous transmission paths is constructed, and a path quality map is generated, including: Based on the storage preferences in the metadata tags, multiple transmission paths from the source node to the backup storage node are defined, and a performance model for heterogeneous transmission paths is constructed. Based on the performance model of the heterogeneous transmission path, the response time of the path medium characteristics is predicted according to the storage medium characteristics and real-time queue requests in the storage preference. At the same time, considering the transmission success rate and latency fluctuation of the transmission path, the network stability is predicted. Based on the response time and network stability of the obtained path medium characteristics, a path quality map is constructed.

3. The data disaster recovery method supporting multiple backup path switching as described in claim 1, characterized in that, The media matching degree refers to the degree of matching between the data fragmentation I / O mode and the characteristics of the path media.

4. The data disaster recovery method supporting multiple backup path switching as described in claim 1, characterized in that, Monitoring the performance of allocating the optimal transmission path includes: The performance of allocating the optimal transmission path is determined by the standardized deviation of the accumulated transmission delay. When an anomaly is detected, the data fragments are migrated to the new optimal transmission path based on the path quality map.

5. The data disaster recovery method supporting multiple backup path switching as described in claim 1 or 4, characterized in that, Based on the monitoring results, incremental integrity verification is performed on the recovered data shards, including: The data shards in the backup storage are located based on the data shard identifier in the metadata tag. The contents of the data shards are read, a Merkle proof is generated for each data shard whose data contents are read, and the Merkle root hash is written into the lightweight blockchain.

6. The data disaster recovery method supporting multiple backup path switching as described in claim 4, characterized in that, Merkle proofs generated using metadata tags and stored on the blockchain are used for parallel verification, and data fragments that failed verification are retransmitted, including: The thread pool is used to verify each data fragment of the read data content in parallel, and the data fragment identifier is matched with the generated Merkle proof. If there are data fragments that fail to be validated in parallel using a thread pool, retransmission will be triggered based on the recovery priority in the metadata tag.

Images