Data disaster recovery method supporting multi-backup path switching

Abstract

The invention discloses a data disaster recovery method supporting multi-backup path switching, which comprises the following steps: generating a dynamic granularity data fragment based on the sensitivity of input data and the characteristics of a storage medium, and generating a metadata label containing a data fragment identifier, storage preference and recovery priority; constructing a heterogeneous transmission path performance model through metadata labels, generating a path quality map, distributing an optimal transmission path for each data fragment, and monitoring path performance; detecting anomalies through the standardized deviation of accumulated transmission delay and migrating fragments; carrying out parallel verification by adopting incremental integrity verification and Merkle proof stored in the block chain, and preferentially retransmitting fragments which fail in verification; according to the invention, through the cooperation of dynamic data fragmentation, backup path optimization and recovery verification, the reliability and stability of data disaster recovery are ensured so as to meet the actual business requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of data disaster recovery, and in particular to a data disaster recovery method supporting multi-backup path switching. Background Art

[0002] With the rapid development of information technology, data has become a critical asset for businesses and organizations, and data disaster recovery technology has gradually become a crucial means of ensuring data security and business continuity. Traditional data disaster recovery methods typically use a single backup path for data transmission and storage, with fixed data sharding granularity. These methods fail to account for differences in data sensitivity, storage media characteristics, and network path performance. As a result, in practical applications, they struggle to meet the diverse requirements for data recovery speed and reliability in different business scenarios. In recent years, with the rise of cloud computing, distributed storage, and blockchain technologies, data disaster recovery technology has gradually developed towards refinement, intelligence, and efficiency.

[0003] However, existing data disaster recovery methods still have some shortcomings. First, traditional methods usually only consider the sensitivity differences of the data itself and the performance characteristics of the storage medium, resulting in a lack of pertinence in the data sharding granularity and path selection strategy, making it difficult to achieve optimal performance in data backup and recovery. Second, traditional methods often ignore the real-time performance fluctuations of the transmission path and the response characteristics of the storage medium during the path selection process, and fail to establish an effective path quality assessment model, resulting in blindness in path selection. Third, traditional methods lack an integrity verification mechanism during the data recovery process, especially the failure to effectively utilize blockchain technology for parallel verification and fast retransmission, resulting in a long data recovery process and difficulty in meeting the needs of rapid data recovery. Therefore, there is an urgent need for a data disaster recovery method that can take into account data sensitivity, storage medium characteristics, and transmission path performance, and supports switching of multiple backup paths.

[0004] CN118152186A discloses a data recovery method and apparatus in a distributed object storage system. This method primarily utilizes an object recovery component to restore backup data from a tape library to a distributed file system, obtaining the entire contents of the restored object. By scanning the entire contents of the restored object, metadata for each object is parsed and written to a distributed database. Subsequently, upon receiving an object retrieval request initiated by a client, the corresponding object is retrieved from the distributed file system based on the metadata. While metadata is mentioned, the solution does not further consider it, failing to account for differences in data sensitivity and the performance characteristics of the storage medium.

[0005] CN117149526A discloses a data recovery method, system, and device for an MPP database. The method primarily utilizes a client to initiate a data backup request, import the information to be backed up, connect to the MPP database, read metadata information, retrieve the MPP database backup table, metadata information, and the HDFS location where the data file is stored, back up the data file from HDFS to object storage, then select the corresponding backup table, obtain the backup file's path on the object storage using the recorded backup information, call the object storage API to obtain the specified path on the object storage, and restore the file to the database on the MPP database. This solution primarily backs up to the object storage file system rather than the storage medium. While this solution takes the backup path into account, it lacks consideration for the storage medium, making it difficult to meet the data recovery speed requirements of different business scenarios. Summary of the Invention

[0006] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0007] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a data disaster recovery method supporting multi-backup path switching to solve the problems raised in the background technology.

[0008] To solve the above technical problems, the present invention provides the following technical solution: a data disaster recovery method supporting multi-backup path switching, comprising:

[0009] Obtain input data, generate dynamic granularity data shards based on the sensitivity of the input data and storage medium characteristics, and generate metadata tags for each data shard;

[0010] By using the metadata tags, a performance model of the heterogeneous transmission path is constructed to generate a path quality map;

[0011] Allocate an optimal transmission path for each data shard based on the metadata tag and the path quality map, and monitor the performance of the allocated optimal transmission path;

[0012] Based on the monitoring results, the restored data shards are incrementally verified for integrity, and the Merkle proof generated by the metadata tag and stored in the blockchain is used for parallel verification. The data shards that failed the verification are retransmitted, thereby achieving efficient backup and priority recovery of the data shards.

[0013] As a preferred solution of the data disaster recovery method supporting multiple backup path switching described in the present invention, the method includes: obtaining input data, generating data slices with dynamic granularity based on the sensitivity of the input data and storage medium characteristics, and generating metadata tags for each data slice, including:

[0014] Generate a sensitivity score based on the entropy density, access pattern ratio and structural correlation of the input data;

[0015] Calculate the compatibility between data shards and storage media based on the sequential read bandwidth, random write latency, and maximum IOPS of the storage media;

[0016] The data sharding granularity is determined according to the sensitivity score and the adaptability, and a metadata tag including a data sharding identifier, a storage preference, and a recovery priority is generated.

[0017] As a preferred solution of the data disaster recovery method supporting multiple backup path switching according to the present invention, it further includes:

[0018] The data shard identifier in the metadata tag is generated using a hash value, the storage preference in the metadata tag is determined by the storage medium type and the data shard I / O mode, and the recovery priority in the metadata tag is calculated based on the sensitivity score and the business SLA coefficient.

[0019] As a preferred solution of the data disaster recovery method supporting multi-backup path switching described in the present invention, wherein: using the metadata tags, a performance model of the heterogeneous transmission path is constructed to generate a path quality map, including:

[0020] Based on the storage preferences in the metadata tags, multiple transmission paths are defined from the source node to the backup storage node, and a performance model of the heterogeneous transmission paths is constructed;

[0021] The performance model of the heterogeneous transmission path is used to predict the response time of the path medium characteristics based on the storage medium characteristics and real-time queue requests in the storage preference. At the same time, the network stability is predicted by considering the transmission success rate and delay fluctuation of the transmission path.

[0022] A path quality map is constructed based on the response time and network stability of the obtained path medium characteristics.

[0023] As a preferred solution of the data disaster recovery method supporting multi-backup path switching described in the present invention, wherein: based on the metadata label and the path quality map, the optimal transmission path is allocated to each data slice, including:

[0024] Based on the storage preference and recovery priority in the metadata tags and the resulting path quality map, a multi-dimensional decision matrix including media matching, network stability, and recovery priority is constructed.

[0025] According to the multidimensional decision matrix, an optimal transmission path is allocated to each data slice, and the allocation of the optimal transmission path to each data slice is used as a constrained optimization problem, and the constrained optimization problem is solved by an improved Hungarian algorithm.

[0026] As a preferred solution of the data disaster recovery method supporting multiple backup path switching described in the present invention, the medium matching degree is the matching degree between the data slice I / O mode and the path medium characteristics.

[0027] As a preferred solution of the data disaster recovery method supporting multi-backup path switching described in the present invention, the improved Hungarian algorithm includes:

[0028] The medium matching degree, network stability and restoration priority in the multi-dimensional decision matrix are converted into a single weight matrix through weighted summation, and transmission path load, restoration time and resource constraints are established.

[0029] As a preferred solution of the data disaster recovery method supporting multiple backup path switching according to the present invention, monitoring the performance of the allocated transmission path includes:

[0030] The normalized deviation of the accumulated transmission delay is used to determine whether the performance of the allocated transmission path is abnormal. When the performance of the allocated transmission path is abnormal, the data fragments are migrated to a new transmission path based on the path quality map.

[0031] As a preferred solution of the data disaster recovery method supporting multi-backup path switching described in the present invention, wherein: based on the monitoring results, incremental integrity verification is performed on the recovered data slices, including:

[0032] Locate the data shards in the backup storage based on the data shard identifier in the metadata tag, read the data shard content, generate a Merkle proof for each data shard whose data content was read, and write the Merkle root hash to the lightweight blockchain;

[0033] As a preferred solution of the data disaster recovery method supporting multi-backup path switching described in the present invention, wherein: using metadata tags to generate and store Merkle proofs in the blockchain for parallel verification, and retransmitting data fragments that failed verification, the method includes:

[0034] Use a thread pool to parallelly verify each data shard that reads the data content, matching the data shard identifier with the generated Merkle proof;

[0035] If there are data shards that fail parallel verification using the thread pool, retransmission is triggered based on the recovery priority in the metadata tag.

[0036] Compared with the prior art, the invention has the following beneficial effects:

[0037] 1. Dynamically granular data sharding is generated by inputting data sensitivity and storage medium characteristics, and combined with the storage preferences and recovery priorities in metadata tags, efficient adaptation of data sharding to storage media is achieved. Compared with traditional methods, this invention can dynamically adjust the data sharding granularity based on data characteristics and business needs, thereby improving data backup efficiency.

[0038] 2. Using an improved Hungarian algorithm for multi-objective optimization, the optimal transmission path is assigned to each data shard. Simultaneously, transmission path performance is monitored in real time and shards are dynamically migrated to new transmission paths when anomalies are detected. This effectively addresses network fluctuations and storage media performance degradation, ensuring stable and reliable data transmission and overcoming the limitations of traditional path selection that can easily lead to data backup failures.

[0039] 3. An incremental integrity verification mechanism is adopted to perform parallel verification only on the recovered shards. The data shard identifier in the metadata tag is matched with the Merkle proof stored in the lightweight blockchain. Compared with traditional verification methods, this invention can quickly verify and restore data. At the same time, through the storage of lightweight blockchain, the verification results are guaranteed to be tamper-proof and traceable. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0041] Figure 1 The figure is an overall flow chart of a data disaster recovery method supporting multi-backup path switching according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0045] The present invention is described in detail with reference to schematic diagrams. For ease of illustration, cross-sectional views of device structures may be partially enlarged and not to scale when describing embodiments of the present invention. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0046] In the description of the present invention, it should be noted that the terms "upper, lower, inner, and outer" and other references to orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0047] In this disclosure, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be interpreted broadly. For example, they may refer to fixed, removable, or integral connections. They may also refer to mechanical, electrical, or direct connections, indirect connections through an intermediary, or internal communication between two components. Those skilled in the art will understand the specific meanings of these terms in this disclosure.

[0048] Example 1

[0049] Reference Figure 1 , which is the first embodiment of the present invention, provides a data disaster recovery method supporting multi-backup path switching, including:

[0050] S1. Obtain input data, generate dynamic granularity data shards based on the sensitivity of the input data and storage medium characteristics, and generate metadata tags for each data shard;

[0051] Specifically, input data refers to the raw data that needs to be backed up and recovered, including database data (such as relational database tables, NoSQL documents, key-value stores), file system data (such as documents, log files, configuration files), multimedia data (such as images, videos), and real-time data (such as log streams, sensor data, transaction records);

[0052] Furthermore, a sensitivity score is generated based on the entropy density, access pattern ratio and structural correlation of the input data;

[0053] Specifically, the randomness of the data is calculated based on the Shannon entropy formula to obtain the entropy density E of the data:

[0054] E=―∑p(a i )logP(a I )

[0055] Among them, P(a i ) represents the byte value a in the data I (range 0 to 255) probability of occurrence;

[0056] It should be noted that the entropy density of data reflects the randomness and information content of the data. The unit is bits / bytes. The higher the value, the more sensitive information the data contains (such as encrypted data). For example, encrypted databases often have a higher entropy density than log files.

[0057] Specifically, the access mode ratio refers to the ratio of data in read / write operations. It is dimensionless and has a value range of [0, 1]. Higher values ​​indicate frequent data access and require priority backup. For example, for sequential reads and random writes, frequently accessed data is usually treated as sensitive information (such as account numbers and passwords) and requires priority backup and recovery. In actual processing, the access mode ratio is calculated using a sliding window to calculate the ratio of data in read / write operations:

[0058]

[0059] Where, wn represents the number of data write operations; rn represents the number of data read operations;

[0060] Specifically, structural correlation refers to the organization or dependency relationship between data. It is dimensionless and has a value range of [0, 1]. Higher values ​​indicate stronger data correlation and require a complete backup. For example, database table foreign keys and file system directory structures require a complete backup for tables with strong correlations (primary and secondary tables).

[0061] It should be explained that, because the organization and dependency relationships between data are very complex in practical applications, it is necessary to use existing graph theory methods to identify which data tables have strong associations. Graph theory methods can abstract tables into nodes and foreign key associations into edges. Edge weights are assigned by the frequency of fields in the table or the amount of data interaction between tables to form a directed graph. The directed graph is processed to identify data tables with strong associations. Then, by calculating the proportion of data tables with strong associations in all data tables with associations (including strong and weak associations), the quantified structural association degree can be obtained.

[0062] Specifically, in order to unify the dimensions and balance the influence of various factors, the sensitivity score is calculated using weighted summation. The resulting sensitivity score formula is expressed as:

[0063] S=ω1·norm(E)+ω2·A+ω3·R scd

[0064] Among them, S represents the sensitivity score. The higher the value, the higher the data sensitivity, the more fine-grained sharding and higher recovery priority are required. The value range is [0,1]. A represents the access mode ratio. R scd It is expressed as structural correlation; v1, ω2, and ω3 represent the sensitivity score, access mode ratio, and weight coefficient of structural correlation respectively; norm represents the normalization operation. The normalization operation in the solution of the present invention adopts Min-Max normalization.

[0065] For example, for an encrypted database, assuming norm(E)=0.9375, A=0.8, R scd =0.9, and the weight coefficients are 0.4 and 0.3 respectively. 0.3 is the classic value (such as obtained through business scenario analysis or machine learning optimization), and we get:

[0066] S=0.4·0.9375+0.3·0.8+0.3·0.9=0.885

[0067] Conclusion: This sensitivity score indicates that the data is highly sensitive and requires finer-grained sharding and a higher recovery priority.

[0068] It should be noted that the sensitivity score shows the sensitivity of the data and also determines the granularity of subsequent data sharding and the priority of data recovery;

[0069] Furthermore, the compatibility between the data shards and the storage medium is calculated based on the sequential read bandwidth, random write latency, and maximum IOPS of the storage medium.

[0070] It should be explained that storage media characteristics refer to the performance parameters of storage media (such as NVMe SSDs and SATA HDDs). These performance parameters include the sequential read bandwidth, random write latency, and maximum IOPS of the storage media.

[0071] Specifically, the sequential read bandwidth of a storage medium refers to the maximum data transfer rate (MB / s or GB / s) when the storage medium continuously reads data. This sequential read bandwidth is obtained from the storage manufacturer's specifications. Its main function is to quickly transfer data using sequential read bandwidth when backing up large files (such as database snapshots).

[0072] Specifically, random write latency refers to the average time (μs or ms) required for the storage medium to complete a random write operation, which is mainly used to constrain the performance of the data shard transmission path;

[0073] Specifically, maximum IOPS (Input / Output Operations Per Second) refers to the maximum number of read and write operations that the storage medium can handle per second. Its main function is to enable the parallel transmission of multiple data shards during backup processing. In addition, it can also verify the integrity of multiple data shards during data recovery.

[0074] Specifically, according to the order of the above storage media, the bandwidth R seq , random write latency W rand and maximum IOPS, define the storage medium characteristic vector M:

[0075] M=(R seq ,W rand ,IOPS max )

[0076] Among them, IOPS max Expressed as maximum IOPS;

[0077] Specifically, the compatibility between data shards and storage media is calculated using the storage medium characteristic vector:

[0078]

[0079] Among them, F is the adaptability of data shards and storage media, indicating the degree of matching between storage media and data shards. The higher the value, the more suitable it is for allocation. r ·S·norm(R seq ) reflects the need for high-bandwidth storage due to data sensitivity and is suitable for large file backup. s Expressed as the weight coefficient of the item; It reflects the storage medium's ability to support random access and is suitable for parallel processing of small data fragments.m Expressed as the weight coefficient of the item;

[0080] Furthermore, the data sharding granularity is determined based on the sensitivity score and adaptability, and metadata tags are generated that include data shard identification, storage preference, and recovery priority;

[0081] Specifically, the determined data sharding granularity N is expressed as:

[0082]

[0083] Among them, D total It represents the total data capacity, and scale is the scaling factor used to control the number of shards. The default value is 100.

[0084] It should be noted that when implementing multi-path backup transmission, if the data fragment size exceeds the path's Maximum Transmission Unit (MTU), the network layer will fragment the data packet, which will cause CPU and network overhead and also introduce the risk of packet loss. Therefore, it is necessary to set upper and lower limits for determining the data fragment granularity.

[0085] Specifically, the upper limit constraint rule is: the size of a single data fragment does not exceed 90% of the current minimum backup path MTU; the lower limit rule is: each data fragment contains at least 3 complete data units;

[0086] It should be noted that because the MTU includes the protocol header (such as the IP header, TCP header, and application layer header), setting 90% as the upper limit for the size of a data fragment can reserve space for the protocol header and potential additional encapsulation (such as VPN and tunneling protocols). For example, if the MTU is 1500 bytes, the protocol header occupies 40 bytes, and 90% is 1350 bytes, which can ensure that the actual payload of the data fragment fits within the protocol header.

[0087] Furthermore, the data shard identifier in the metadata tag is generated using a hash value, the storage preference is determined by the storage media type and the data shard I / O mode, and the recovery priority in the metadata tag is calculated based on the sensitivity score and the business SLA coefficient;

[0088] Specifically, a hash value of the data shard identifier is generated by the SHA-256 hash algorithm to identify each data shard;

[0089] Specifically, the storage media types in the storage preference include common electronic storage (such as USB flash drives and SD cards), magnetic storage (such as hard disks and magnetic tapes), and optical storage (such as optical disks);

[0090] It should be noted that, for the data sharding I / O mode, only sequential read and random write situations are considered in the solution of the present invention;

[0091] Furthermore, the data sharding I / O pattern is obtained by calculating the locality index of the access address and the time weight of the access, where the locality index L of the access address is s The calculation formula is as follows:

[0092]

[0093] Among them, addr l It represents the address of the first access, n is the number of accesses, and max(addr) represents the maximum access address;

[0094] It should be noted that if the last accessed address (previous address) does not exist, it is ignored. Usually, the first address does not have the last accessed address (previous address);

[0095] Specifically, the locality index indicates the concentration of access addresses, where low values ​​indicate high locality (random access, small address changes) and high values ​​indicate low locality (sequential access, large address jumps);

[0096] Furthermore, the time weight of the visit L t The calculation formula is as follows:

[0097]

[0098] Among them, λ represents the access time decay factor, hit(l) represents whether the lth access is effective (the value is 0 or 1), t current Represents the current access time, t l It is expressed as the time of the first visit;

[0099] Specifically, the access time weight reflects the frequency of data access;

[0100] It should be noted that, through the above-mentioned access time weight and the locality index of the access address, the number of accesses and the access time of the current storage medium type can be obtained, thereby determining the storage preference during backup;

[0101] For example, assume that the access log of the database is analyzed and the latest 10 access records (in reality, there may be hundreds of access records) are collected. The access log includes the access address and access time. The access records are shown in Table 1 below:

[0102] Table 1

[0103] Access number (l) <![CDATA[Access address (addr l )]]> <![CDATA[Access time (t l )]]> Is it effective (hit(l)) 1 100 0 1 2 102 1 1 3 101 2 1 4 200 3 1 5 201 4 1 6 300 5 0 7 103 6 1 8 104 7 1 9 105 8 1 10 106 9 1

[0104] Parameter settings: max(addr) = 10,000KB (assuming the total database size is 10GB), n = 10, λ = 0.9, t current=10, the data unit is 1KB (each record);

[0105] According to the locality index L of the access address s , calculate the address difference of each access:

[0106] Access sequence number 1: no previous address, ignored;

[0107] Access number 2: |addr2―addr 2―1 |=|102―100|=2KB

[0108] Access number 3: |addr3―addr 3―1 |=|101―102|=1KB

[0109] Access number 4: |addr4―addr 4―1 |=|200―101|=99KB

[0110] Access number 5: |addr5―addr 5―1 |=|201―200|=1KB

[0111] Access number 6: |addr6―addr 6―1 |=|300―201|=99KB

[0112] Access number 7: |addr7―addr 7―1 |=|103―300|=197KB

[0113] Access sequence number 8: |addr8―addr 8―1 |=|104―103|=1KB

[0114] Access sequence number 9: |addr9―addr 9―1 |=|105―104|=1KB

[0115] Access number 10: |addr 10 ―addr 10―1 |=|106―105|=1KB

[0116] Remove access number 1, normalize the difference, and divide by max(addr) to get:

[0117] Access number 2: 2 / 10,000 = 0.0002;

[0118] Access number 3: 1 / 10,000 = 0.0001;

[0119] Access number 4: 99 / 10,000=0.0099;

[0120] Access number 5: 1 / 10,000 = 0.0001;

[0121] Access number 6: 99 / 10,000=0.0099;

[0122] Access number 7: 197 / 10,000 = 0.0197;

[0123] Access number 8: 1 / 10,000 = 0.0001;

[0124] Access number 9: 1 / 10,000 = 0.0001;

[0125] Access number 10: 1 / 10,000 = 0.0001;

[0126] Calculate the average value after accumulation get:

[0127] 0.0002+0.0001+0.0099+0.0001+0.0099+0.0197+0.0001+0.0001+0.0001=0.0402

[0128] 0.0402 / 9≈0.0045 (keep 4 decimal places)

[0129] As you can see, the locality index is low (0.0045), indicating that the access address changes little and the access pattern is random, which is suitable for high IOPS storage media;

[0130] According to the time weight L of the visit t , calculate the weight of each visit (keep 4 decimal places):

[0131] Access sequence 1: t1 = 0, λ = 1, 0.9 10―0 ×1=0.9 10 ≈0.3487

[0132] Access number 2: t1 = 1, λ = 1, 0.9 10―1 ×1=0.9 9 ≈0.3874

[0133] Access number 3: t1 = 2, λ = 1, 0.9 10―2 ×1=0.9 8 ≈0.4305

[0134] Access number 4: t1 = 3, λ = 1, 0.9 10―3 ×1=0.9 7 ≈0.4783

[0135] Access number 5: t1 = 4, λ = 1, 0.910―4 ×1=0.9 6 ≈0.5314

[0136] Access number 6: t1 = 5, λ = 0, 0.9 10―5 ×0=0

[0137] Access number 7: t1 = 6, λ = 1, 0.9 10―6 ×1=0.9 4 =0.6561

[0138] Access number 8: t1 = 7, λ = 1, 0.9 10―7 ×1=0.9 3 =0.7290

[0139] Access number 9: t1 = 8, λ = 1, 0.9 10―8 ×1=0.9 2 =0.8100

[0140] Access number 10: t1 = 9, λ = 1, 0.9 10―9 ×1=0.9 1 =0.9000

[0141] Summing to get

[0142] 0.3487+0.3874+0.4305+0.4783+0.5314+0.6561+0.7290+0.8100+0.9000≈5.2714. As can be seen, because the access time weight accounts for more than half (the upper limit of the time weight is 10), it indicates that access is frequent and recent access is the main access mode (the weight increases from 0.3487 to 0.9). Therefore, a storage medium with high frequency access is required.

[0143] Based on the locality index of the access address and the access time weight, we can determine the storage preference. Specifically, we choose storage media with high IOPS, suitable for random reads and writes, and high access frequency (such as Alibaba Cloud ESSD and NVMe SSD).

[0144] Specifically, the business SLA coefficient refers to the contract between the enterprise and the service provider or internal IT system, which specifies the performance indicators of the service (such as recovery time objective, recovery point objective, and availability), usually specified by the system administrator or the business party;

[0145] Specifically, the restoration priority P is determined based on the service SLA coefficient and sensitivity score:

[0146] P=S×SLA coeff

[0147] Among them, SLAcoeff Expressed as business SLA coefficient;

[0148] S2. Build a performance model of heterogeneous transmission paths through metadata tags and generate a path quality map.

[0149] Furthermore, based on the storage preferences in the metadata tags, multiple transmission paths from the source node to the backup storage node are defined to build a performance model for heterogeneous transmission paths;

[0150] Among them, each transmission path connects to a different storage medium and has different network characteristics;

[0151] It should be explained that, in the solution of the present invention, the transmission path refers to the data transmission channel from the source node to the backup storage node. These paths are heterogeneous (i.e., have different network characteristics) and are used to transmit data shards to the backup storage node;

[0152] Furthermore, through the performance model of heterogeneous transmission paths, the response time of the path medium characteristics is predicted based on the storage medium characteristics and real-time queue requests in the storage preference. At the same time, the transmission success rate and delay fluctuation of the transmission path are considered to predict the network stability.

[0153] Specifically, the response time of the path medium characteristic refers to the storage medium response time of the backup storage node at the end point of the path; the path medium characteristic refers to the performance of the storage medium involved in the transmission path;

[0154] Specifically, real-time queue requests refer to the current I / O queue depth and request type (read / write, sequential / random). The current I / O queue depth refers to the number of I / O requests currently being processed by the backup storage node.

[0155] Specifically, the response time T of the predicted path medium characteristics med :

[0156]

[0157] Where β is the bandwidth attenuation coefficient (dimensionless, default value is 0.5, adjusted according to the storage medium type, such as 0.3 for NVMeSSD and 0.7 for HDD); N req is the number of data shard requests, that is, the number of shards to be transferred; ql is the number of pending requests for the storage medium, that is, the current I / O queue depth, which is obtained in real time through the storage system API (such as Linux / proc / diskstats); Expressed as transmission delay due to bandwidth limitations; Represents processing delays due to queue depth and random write latency;

[0158] For example, assuming β = 0.5, N req =100, R seq =500MB / s,ql=50,W rand =0.1,IOPS max =100000, we get:

[0159]

[0160] Conclusion: The response time of 0.10005ms indicates the efficiency of the storage medium in handling the current load;

[0161] It should be explained that bandwidth can improve the storage medium's throughput, thereby enhancing the storage medium's ability to respond to requests, and IOPS determines the storage medium's response speed under high load. Therefore, both can affect the storage medium's maximum responsiveness. In addition, high latency means that the storage medium responds slowly to requests, while low latency means that the storage medium responds quickly. Therefore, for small requests (such as 4KB), low latency is more dependent on low latency because the overhead of each operation is high. For large requests (such as 1MB), bandwidth is more dependent, and latency has less impact.

[0162] Specifically, predict network stability S index :

[0163]

[0164] Where success_rate is the weighted average success rate. If 9 out of 10 transmissions are successful, then success_rate = 0.9. It is expressed as the latency fluctuation coefficient, which reflects network jitter. The larger the fluctuation coefficient, the worse the stability. μ(latency) is expressed as the mean latency. σ(latency) is expressed as the standard deviation of latency.

[0165] Specifically, a path quality map is formed based on the response time and network stability of the obtained path medium characteristics;

[0166] It should be noted that the path quality map includes the path (path_id), storage medium type (media_type), response time of the path medium characteristics (T_media), and network stability (Stability_index). For example, the code assignment is: [{"path_id":1,"media_type":"NVMe SSD","T_media":1.02,"Stability_index":0.99}];

[0167] S3, based on metadata tags and path quality maps, assigns the optimal transmission path to each data shard and monitors the performance of the assigned optimal transmission path;

[0168] Furthermore, based on the storage preference and recovery priority in the metadata tags and the resulting path quality map, a multidimensional decision matrix including media matching, network stability, and recovery priority is constructed.

[0169] Specifically, medium matching M atc h med,ij The degree of matching between the data shard I / O pattern and the path medium characteristics is calculated using cosine similarity as follows:

[0170]

[0171] in, Represented as a data shard I / O pattern vector, g1 is the sequential read ratio (based on access pattern ratio A), g2 is the random write ratio (1-A); Expressed as the path medium characteristic vector, cos(θ) represents the cosine similarity; ‖·‖ represents the norm of the vector, that is, the length or size of the vector;

[0172] For example, assuming that the access mode ratio of data shards is A=0.8, then Path get:

[0173]

[0174] (Keep four decimal places)

[0175] (Keep four decimal places)

[0176] (Keep four decimal places)

[0177] Conclusion: 0.2907 indicates that the matching degree between data shards and path media is low, and path allocation needs to be further optimized;

[0178] Specifically, each item in the multidimensional decision matrix represents the comprehensive score of assigning a data slice i to a path j;

[0179] Furthermore, according to the multi-dimensional decision matrix, the optimal transmission path is assigned to each data shard, and the assignment of the optimal transmission path to each data shard is formulated as a constrained optimization problem, which is solved by the improved Hungarian algorithm;

[0180] It should be explained that the Hungarian algorithm is a classic bipartite graph matching algorithm, mainly used to solve allocation problems. In the application scenario of the present invention, its principle is to find the maximum weight and match it so that each data shard can be assigned to a unique path with the maximum total weight (score). The traditional Hungarian algorithm assumes one-to-one matching and cannot directly handle multi-objective optimization and does not support complex constraints. In order to handle the multi-objective optimization problem in the solution of the present invention, it needs to be improved.

[0181] Specifically, the improved Hungarian algorithm is as follows:

[0182] The medium matching degree, network stability and recovery priority in the multi-dimensional decision matrix are converted into a single weight matrix through weighted summation, and the transmission path load, recovery time and resource constraints are established;

[0183] It should be noted that the purpose of converting to a single weight matrix is ​​to convert the multi-objective optimization problem into a single-objective optimization problem;

[0184] Specifically, a single weight matrix M ij Expressed as:

[0185]

[0186] in, Expressed as the weight of medium matching, ω P Expressed as the weight of network stability, ω S expressed as the weight of the restoration priority;

[0187] Specifically, the single-objective optimization problem is expressed as:

[0188]

[0189] Specifically, a transmission path load constraint is established, that is, each path can carry a maximum of a limited number of high-priority fragments:

[0190]

[0191] Among them, y ij Indicates that the data shard is a high-priority shard and is assigned to path j; It is expressed as arbitrary;

[0192] Specifically, establish a recovery time constraint, that is, the total recovery time T total Meeting the Recovery Time Objective (RTO):

[0193]

[0194] Among them, x ij Represented as data shards;

[0195] Specifically, establish resource constraints, that is, bandwidth and IOPS must not exceed the limit:

[0196]

[0197] It should be noted that the improved Hungarian algorithm allocates the optimal transmission path to each data shard, which can ensure the stability of backup data during transmission;

[0198] It should be noted that if the performance of the allocated transmission path is abnormal, it may cause data shard transmission failure or delay, thereby affecting the integrity of the backup and increasing the risk of data loss during disaster recovery. Therefore, it is necessary to make a judgment on this.

[0199] Furthermore, the normalized deviation of the accumulated transmission delay is used to determine whether the performance of the assigned transmission path is abnormal. When the performance of the assigned transmission path is abnormal, the data fragment is migrated to a new transmission path based on the path quality map.

[0200] Specifically, transmission delay data is collected once per second and stored in a time window. The cumulative transmission delay is calculated by summing all delays within the time window. The standardized deviation of the cumulative transmission delay is then calculated, and an abnormal threshold is set based on expert experience (for example, if the standardized deviation of the cumulative transmission delay is greater than 2, the performance fluctuation exceeds 2 times the mean). If the abnormal threshold is exceeded, the transmission path is marked as abnormal, the data segment transmission on the abnormal path is suspended, the current abnormal path (path_id) is recorded, and the abnormal path (path_id) is removed based on the path (path_id) in the path quality map, and the metadata label is updated. The path with the highest score in the multidimensional decision matrix is ​​recalculated as the new path, and the data segment is migrated to the new path for transmission.

[0201] S4. Based on the monitoring results, incremental integrity verification is performed on the recovered data shards. Merkle proofs generated by metadata tags and stored in the blockchain are used for parallel verification. Data shards that fail verification are retransmitted, thereby achieving efficient backup and priority recovery of data shards.

[0202] Furthermore, based on the data shard identifier in the updated metadata tag, the data shard in the backup storage is located, the data shard content is read, a Merkle proof is generated for each data shard whose data content is read, and the Merkle root hash is written into the lightweight blockchain (private chain);

[0203] It should be explained that a Merkle proof is a data structure based on a Merkle tree (a binary hash tree), which is used in the solution of the present invention to prove whether a data shard belongs to a set of data shards and to verify its integrity and consistency.

[0204] Specifically, the updated data shard identifier is used as the key to query the backup storage, obtain the content of the data shard, verify whether the data shard exists, and if not, mark it as a failure. If it exists, generate a Merkle proof, calculate the hash value of each data shard content, and the hash value is consistent with the initial hash value of the data shard identifier generation method (i.e., using SHA-256 hash) to obtain the shard hash value; use the shard hash value as the leaf node, calculate the hash value for each pair (leaf node and leaf node) layer by layer, until traversing to the root node, obtain the Merkle root hash, and write the Merkle root hash into the lightweight blockchain;

[0205] It should be noted that the immutability of blockchain ensures that the Merkle root hash of data shard verification is credible to prevent data from being maliciously modified. In addition, blockchain can also provide timestamps and historical records to facilitate data traceability.

[0206] Furthermore, a thread pool is used to parallelize the verification of each data shard that reads the data content, matching the data shard identifier with the generated Merkle proof;

[0207] Specifically, a fixed-size thread pool is created (e.g., 8 threads, based on the number of CPU cores), a thread-safe queue (e.g., Python's concurrent.futures.ThreadPoolExecutor) is used to allocate a thread to each data shard, and the Merkle root hash is reconstructed. If the reconstructed Merkle root hash is consistent with the Merkle root hash written into the blockchain, the verification is successful; otherwise, the verification fails.

[0208] Furthermore, if there are data shards that fail parallel verification using the thread pool, retransmission is triggered based on the recovery priority in the metadata tag;

[0209] Specifically, the data shards that failed verification and their metadata tags are sorted according to the recovery priority, and steps S2 to S4 are re-executed (during this period, the data shard status in the backup storage is updated) until the retry limit (e.g., 3 times) is reached;

[0210] It should be noted that by retransmitting only the data fragments that failed verification, other data fragments are not affected, repeated transmission of data fragments is avoided, and the rapid recovery requirements of data disaster recovery are met.

[0211] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0212] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0213] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0215] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0216] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data disaster recovery method supporting multi-backup path switching, characterized in that: include: Obtain input data, generate dynamic granularity data shards based on the sensitivity of the input data and storage medium characteristics, and generate metadata tags for each data shard; By using the metadata tags, a performance model of the heterogeneous transmission path is constructed to generate a path quality map; Allocate an optimal transmission path for each data shard based on the metadata tag and the path quality map, and monitor the performance of the allocated optimal transmission path; Based on the monitoring results, the restored data shards are incrementally verified for integrity, and the Merkle proof generated by the metadata tag and stored in the blockchain is used for parallel verification. The data shards that failed the verification are retransmitted, thereby achieving efficient backup and priority recovery of the data shards.

2. The data disaster recovery method supporting multiple backup path switching according to claim 1, characterized in that: Obtain input data, generate dynamic granularity data shards based on the sensitivity of the input data and storage medium characteristics, and generate metadata tags for each data shard, including: Generate a sensitivity score based on the entropy density, access pattern ratio and structural correlation of the input data; Calculate the compatibility between data shards and storage media based on the sequential read bandwidth, random write latency, and maximum IOPS of the storage media; The data sharding granularity is determined according to the sensitivity score and the adaptability, and a metadata tag including a data sharding identifier, a storage preference, and a recovery priority is generated.

3. The data disaster recovery method supporting multiple backup path switching according to claim 2, wherein: Also includes: The data shard identifier in the metadata tag is generated using a hash value, the storage preference in the metadata tag is determined by the storage medium type and the data shard I / O mode, and the recovery priority in the metadata tag is calculated based on the sensitivity score and the business SLA coefficient.

4. The data disaster recovery method supporting multiple backup path switching according to claim 2, wherein: Using the metadata tags, a performance model of heterogeneous transmission paths is constructed to generate a path quality map, including: Based on the storage preferences in the metadata tags, multiple transmission paths are defined from the source node to the backup storage node, and a performance model of the heterogeneous transmission paths is constructed; The performance model of the heterogeneous transmission path is used to predict the response time of the path medium characteristics based on the storage medium characteristics and real-time queue requests in the storage preference. At the same time, the network stability is predicted by considering the transmission success rate and delay fluctuation of the transmission path. A path quality map is constructed based on the response time and network stability of the obtained path medium characteristics.

5. The data disaster recovery method supporting multiple backup path switching according to claim 4, characterized in that: Based on the metadata tags and the path quality map, an optimal transmission path is assigned to each data shard, including: Based on the storage preference and recovery priority in the metadata tags and the resulting path quality map, a multi-dimensional decision matrix including media matching, network stability, and recovery priority is constructed. According to the multidimensional decision matrix, an optimal transmission path is allocated to each data slice, and the allocation of the optimal transmission path to each data slice is used as a constrained optimization problem, and the constrained optimization problem is solved by an improved Hungarian algorithm.

6. The data disaster recovery method supporting multiple backup path switching according to claim 5, characterized in that: The media matching degree is the matching degree between the data shard I / O mode and the path media characteristics.

7. The data disaster recovery method supporting multiple backup path switching according to claim 5, characterized in that: The improved Hungarian algorithm includes: The medium matching degree, network stability and restoration priority in the multi-dimensional decision matrix are converted into a single weight matrix through weighted summation, and transmission path load, restoration time and resource constraints are established.

8. The data disaster recovery method supporting multiple backup path switching according to claim 5, characterized in that: Monitoring the performance of the allocated transmission path, including: The normalized deviation of the accumulated transmission delay is used to determine whether the performance of the allocated transmission path is abnormal. When the performance of the allocated transmission path is abnormal, the data fragments are migrated to a new transmission path based on the path quality map.

9. The data disaster recovery method supporting multiple backup path switching according to claim 3 or 8, characterized in that: Based on the monitoring results, incremental integrity verification is performed on the restored data shards, including: The data shards in the backup storage are located according to the data shard identifier in the metadata tag, the data shard contents are read, a Merkle proof is generated for each data shard whose data content is read, and the Merkle root hash is written into the lightweight blockchain.

10. The data disaster recovery method supporting multiple backup path switching according to claim 8, characterized in that: Merkle proofs generated by metadata tags and stored in the blockchain are used for parallel verification, and data shards that fail verification are retransmitted, including: Use a thread pool to parallelly verify each data shard that reads the data content, matching the data shard identifier with the generated Merkle proof; If there are data shards that fail parallel verification using the thread pool, retransmission is triggered based on the recovery priority in the metadata tag.

Images