A data migration method of a database

Abstract

The application relates to the technical field of data migration, and specifically discloses a data migration method for a database, which comprises the following steps: constructing a migration topology graph based on a source storage node cluster, a target storage node cluster and a network device between the two clusters; using a graph neural network model to calculate the expected communication efficiency and the overall topology vulnerability index of each candidate migration link; obtaining the business access log of a data object to be migrated, and performing online streaming time sequence analysis to determine a pre-migration candidate set; determining the available migration throughput according to the current business I / O load; determining the migration object, the migration path and the migration window based on the expected communication efficiency, the overall topology vulnerability index, the pre-migration candidate set and the available migration throughput; dividing the migration object into multiple migration units; and performing data migration on the data object to be migrated. The application can reduce the influence of data migration on online business I / O and reduce the migration verification and rollback range.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, and more particularly to a method for migrating databases. Background Technology

[0002] In cloud computing, distributed storage, object storage, database clusters, and enterprise-level storage systems, data migration is commonly used for load balancing, hot / cold data tiering, storage media replacement, capacity expansion, and fault avoidance. Existing migration solutions typically trigger migration based on node capacity, node load, or data access frequency, and use background threads to copy data from the source storage location to the target storage location.

[0003] As online business scales up, the impact of migration on business I / O becomes more pronounced. If the migration path does not consider link congestion, network device status, and topology vulnerabilities, migration tasks can easily occupy critical links or vulnerable nodes, leading to increased business access latency. If migration scheduling is only executed based on fixed time windows or static bandwidth limits, background migration threads may still compete with business read / write operations for disk queues and network bandwidth. If hot data can only be passively migrated after access volume increases, temporary migrations may overlap with business peaks.

[0004] On the other hand, the migration process also needs to ensure data consistency. Traditional practices often involve calculating data block hashes in real time during the migration process or performing a full verification after the migration, which incurs significant verification overhead. If business writes continue to occur during the migration, relying solely on full snapshots or full rollbacks will expand the scope of conflict handling and increase migration pause times. Summary of the Invention

[0005] The present invention aims to solve the above-mentioned problems. To this end, the present invention provides a database data migration method that can reduce the impact of data migration on online business I / O and reduce the scope of migration verification and rollback.

[0006] This invention provides a database data migration method, the technical solution of which includes: S1: Collect operational status data of the source storage node cluster, the target storage node cluster, and the network devices between them; S2: Construct a migration topology graph using storage nodes and network devices as nodes and the topological connections between storage nodes and network devices as edges; input the running status data into the graph neural network model, obtain the aggregation features after aggregating neighbor information, and calculate the expected communication efficiency and overall topology vulnerability index of each candidate migration link based on the aggregation features; S3: Obtain the business access logs of the data objects to be migrated and perform online streaming time-series analysis. Determine the pre-migration candidate set based on access frequency, first-order growth of access frequency, recent access interval, and object business priority. S4: Continuously monitor the current business I / O load of the data object to be migrated. When the current business I / O load is lower than the business load threshold, execute step S5 and determine the available migration throughput based on the business I / O bandwidth limit, the current business I / O load, the minimum business protection bandwidth, and the migration thread safety factor. S5: Based on expected communication efficiency, overall topology vulnerability index, pre-migration candidate set and available migration throughput, determine migration objects, migration paths and migration windows; S6: Divide the migration object into multiple migration units based on the available migration throughput and migration window; S7: Perform data migration of the data objects to be migrated based on migration unit, migration path, and migration window.

[0007] Furthermore, in step S2, the degree of insufficient remaining space of nodes, the degree of link congestion, the risk of storage queue depth, and the risk of historical link failure are extracted from the aggregated features, and the overall topology vulnerability index is calculated by weighted summation.

[0008] Furthermore, in step S3, online streaming time-series analysis is performed on the business access logs. Within a sliding time window, the access frequency, first-order growth rate of the access frequency, the most recent access interval, and the business priority of the object are statistically analyzed to calculate the popularity prediction score. The calculation formula is as follows: in, The predicted popularity score for data object o; This is the access frequency coefficient; The frequency of access to data object o; This is the coefficient for the increase in access frequency; The first-order increment of the access frequency of data object o; The penalty coefficient for the most recent access interval; The most recent access interval for data object o; To prevent small positive numbers from being divided by zero; This is the business priority coefficient; The business priority of data object o; When the predicted popularity score exceeds the popularity threshold, the corresponding data object is added to the pre-migration candidate set.

[0009] Furthermore, in step S4, the available migration throughput is the non-negative throughput after the upper limit of the service I / O bandwidth is reduced by the migration thread safety factor, and then the current service I / O load and the minimum service protection bandwidth are deducted.

[0010] Furthermore, in step S5, a migration priority score is calculated for each data object and candidate migration path in the pre-migration candidate set. The migration priority score increases with the increase of the heat prediction score, media adaptation benefit and expected communication efficiency, and decreases with the increase of the overall topology vulnerability index and the current service I / O load. Based on the migration priority score, the migration objects and their corresponding migration paths are determined; The migration window is determined by combining the available migration throughput and the preset migration throughput cap.

[0011] Furthermore, the formula for calculating the migration priority score is as follows: in, Score the priority of data object o as it migrates via candidate path r; The predicted popularity score for data object o; Benefits of media adaptation for data object o; The expected communication efficiency of candidate migration path r; This is the overall topological vulnerability index; This is the penalty coefficient for business load. This is the normalized value of the current business I / O load.

[0012] Furthermore, in step S6, the size of the migration unit is dynamically determined based on the length of the migration window and the available migration throughput. The formula for calculating the migration unit size is as follows: in, Size of the migration unit; The maximum amount of data allowed for a single migration unit; Available migration throughput; This is the current migration window length; The migration object is divided into multiple migration units based on the size of the migration unit.

[0013] Furthermore, in step S6, a flexible consistent snapshot layer based on copy-on-write snapshots is established for each migration unit; In step S7, data migration of the data objects to be migrated is performed based on the flexible consistency snapshot layer; The flexible consistency snapshot layer consists of three components: source snapshot version, migration view version, and copy-on-write difference record. The source snapshot version is the data state of the migration unit frozen by performing a copy-on-write snapshot operation on the corresponding source volume of the source storage node before the migration starts. The migration view version is the version identifier of the data view seen by business I / O during the migration process. The copy-on-write difference record is used to record the write operations of the business on the source volume after the migration starts.

[0014] Furthermore, in step S7, when a verification failure occurs after the migration unit is written on the target end, the rollback flexible mechanism of the flexible consistency snapshot layer is triggered. Only the incremental write of the corresponding failed migration unit is canceled, the write-time replication difference record of the migration unit and the corresponding copy on the target end are discarded, and the corresponding failed migration unit is added back to the migration queue.

[0015] Furthermore, before the migration starts, the source volume logical block address range corresponding to each migration unit is pre-scanned using the storage system's idle I / O cycle to generate a pre-computed verification metadata index for each data block of the migration unit. After the migration unit is written on the target side, the pre-computed verification metadata index is used to check the target side data block to reduce real-time full hash calculation during the migration process.

[0016] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. This invention constructs a migration topology graph using storage nodes and network devices as nodes and topology connections as edges. It then inputs runtime status data into a graph neural network model to aggregate neighbor information, obtaining the expected communication efficiency of each candidate migration link and the overall topology vulnerability index. Migration scheduling can thus avoid vulnerable paths with link congestion, queue backlogs, or high historical failure rates, preventing migration tasks from occupying critical links or vulnerable nodes and reducing the impact of the migration process on online service access latency.

[0017] 2. This invention performs online streaming time-series analysis on business access logs within a sliding time window, calculating a popularity prediction score by integrating four factors: access frequency, first-order growth rate of access frequency, most recent access interval, and object business priority. This allows for the early detection of data object popularity trends. Compared to the traditional approach of passively waiting for increased access volume to trigger migration, this invention moves the hot data migration decision forward, effectively avoiding overlap between temporary migrations and business peaks, and reducing competition between migration and business I / O.

[0018] 3. The invention establishes a flexible consistency snapshot layer for each migration unit, consisting of the source snapshot version, the migration view version, and write-time replication difference records, maintaining a data consistency view independently at the migration unit level. When verification fails, a flexible rollback mechanism is triggered, reducing the conflict handling and rollback scope from the entire volume to a single migration unit, significantly shortening migration pause time and improving the fault tolerance and availability of the migration process.

[0019] 4. This invention generates a verification metadata index containing hash values ​​in advance for each data block; after the target end completes the write, it directly compares the index for verification, shifting the hash calculation pressure from the critical migration path to idle periods, significantly reducing the real-time calculation overhead during the migration process, and improving migration throughput while ensuring data correctness.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the method provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0024] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0025] The following is combined with Figure 1 The present invention will be further described in detail below, including a database data migration method of the present invention: In this embodiment, as Figure 1 As shown, a data migration method for a database is provided, including the following steps: S1: Collect the operating status data of the source storage node cluster, the target storage node cluster, and the network devices between them. The operating status data includes the storage medium type, read / write rate, remaining space, and queue depth of the storage nodes, as well as the link bandwidth, link latency, packet loss rate, and historical migration success rate of the network devices.

[0026] This step first identifies the source storage node cluster, target storage node cluster, network devices, as well as the set of data objects to be migrated and the set of candidate migration links. It then reads the metadata directory, cluster management interface, and network control interface of the data objects to be migrated, eliminating unreachable storage nodes, read-only volumes, and data objects without migration permissions. Next, it obtains the storage medium type, read / write speed, remaining space, and queue depth of the storage nodes, as well as the link bandwidth, link latency, packet loss rate, and historical migration success rate of the network devices.

[0027] The candidate migration link set is generated as follows: taking each source storage node in the source storage node cluster as the starting point and each target storage node in the target storage node cluster as the ending point, the candidate migration link set is generated based on all reachable paths in the network topology using the K-shortest path algorithm. Links containing unreachable network devices are removed, and the available bandwidth of the links is pre-screened, removing links with available bandwidth lower than the preset minimum bandwidth threshold.

[0028] S2: Construct a migration topology graph using storage nodes and network devices as nodes and the topological connections between storage nodes and network devices as edges; input the running status data into a graph neural network model, and obtain aggregated features after aggregating neighbor information; calculate the expected communication efficiency and overall topology vulnerability index of each candidate migration link based on the aggregated features. The storage nodes include source storage nodes in the source storage node cluster and target storage nodes in the target storage node cluster.

[0029] In this embodiment, node characteristics include storage medium type, read / write rate, remaining space, and queue depth; edge characteristics include link bandwidth, link latency, packet loss rate, and historical migration success rate. Edge weights are jointly determined by link bandwidth, link latency, packet loss rate, and historical migration success rate, and are calculated using a multi-indicator weighted fusion + nonlinear mapping method.

[0030] In this embodiment, the graph neural network model includes multiple graph convolutional layers, preferably three. Node features, edge features, and adjacency relationships are input into the graph neural network model. Each graph convolutional layer reads its own node representation, neighbor node representation, and link edge weights from the previous layer, aggregates them, and outputs a new node representation. The model output is the aggregated features, i.e., the aggregated node features and edge features.

[0031] The message aggregation process in a graph neural network model is represented as follows: in, The topological representation of node v output by the current graph convolutional layer; For activation functions, the ReLU function is preferred; The transformation matrix is ​​the feature transformation matrix of its own node; Input the topological representation of the current graph convolutional layer for node v; Number the neighboring nodes; Let v be the set of neighboring nodes; Let be the edge weight from node u to node v; The feature transformation matrix of the neighboring nodes; Input the topological representation of the current graph convolutional layer for the neighbor node u; This is the feature transformation matrix of the link edge; Let be the edge features between node u and node v.

[0032] The aggregated features (node ​​features and edge features) are input into an MLP (Multilayer Perceptron) to calculate the expected communication efficiency of each candidate transfer path. The MLP includes two fully connected hidden layers with 128 and 64 neurons in each layer, respectively, using ReLU as the activation function. The output layer is a single-neuron linear output, and the output value is the expected communication efficiency.

[0033] The degree of insufficient remaining space in nodes, link congestion, storage queue depth risk, and historical link failure risk are extracted from the aggregated features. The overall topology vulnerability index is then calculated using a weighted sum. The calculation formula is as follows: in, This is the overall topological vulnerability index; The coefficient for the insufficient node space; The degree of insufficient remaining space for the node; For the link congestion term coefficient; The degree of link congestion; The coefficient for the queue backlog term; Risks associated with storage queue depth; The coefficients for historical fault items; This refers to historical risks associated with link failures. In this embodiment, It is 0.25. It is 0.30. It is 0.20. It is 0.25.

[0034] The higher the overall topology vulnerability index, the greater the risk that the migration task will occupy critical links or vulnerable nodes.

[0035] The specific methods for assessing node remaining space insufficiency, link congestion, storage queue depth risk, and link failure historical risk are as follows: The aggregated node feature vector and aggregated edge feature vector output from the graph neural network are input into four independent multilayer perceptron readout layers. Each readout layer includes a hidden layer (32 neurons) and an output layer (single neuron, Sigmoid activation), outputting the corresponding risk index values, mapped to the [0,1] interval. Specifically, node remaining space insufficiency is obtained by mapping the remaining space-related dimension in the aggregated node features through the first readout layer; link congestion is obtained by mapping the link bandwidth and latency-related dimensions in the aggregated edge features through the second readout layer; storage queue depth risk is obtained by mapping the queue depth-related dimension in the aggregated node features through the third readout layer; and link failure historical risk is obtained by mapping the packet loss rate and historical migration success rate-related dimensions in the aggregated edge features through the fourth readout layer.

[0036] S3: Obtain the business access logs of the data objects to be migrated, and perform online streaming time-series analysis. Determine the pre-migration candidate set based on access frequency, first-order growth of access frequency, recent access interval, and object business priority.

[0037] Online streaming time-series analysis is performed on business access logs. Within a sliding time window, the access frequency, first-order growth rate of access frequency, most recent access interval, and business priority of the statistical objects are collected. After normalization, a popularity prediction score is calculated. The calculation formula is as follows: in, The predicted popularity score for data object o; This is the access frequency coefficient; The frequency of access to data object o; This is the coefficient for the increase in access frequency. The first-order increment of the access frequency of data object o; The penalty coefficient for the most recent access interval; The most recent access interval for data object o; To prevent small positive numbers from being divided by zero; This is the business priority coefficient; The business priority of data object o. Access frequency, first-order increment of access frequency, and most recent access interval are normalized using a maximum-minimum method, with values ​​ranging from [0,1]. The business priority value range is [0,1], obtained by mapping priority tags in the business system metadata, with the highest priority business corresponding to a value of 1 and the lowest priority business corresponding to a value of 0. In this embodiment, It is 0.35. It is 0.25. It is 0.20. It is 0.20.

[0038] When the predicted popularity score exceeds the popularity threshold, the corresponding data object is added to the pre-migration candidate set. Data objects with predicted popularity scores below the popularity threshold are not included in the pre-migration candidate set. If access log sampling is interrupted or the object's business priority is missing, the previous valid popularity state is retained, and migration is prohibited from being triggered solely based on a single access peak. Objects in the pre-migration candidate set are not migrated immediately; instead, they wait until the business I / O load, topology vulnerability, and candidate path status all meet the migration conditions before entering the background migration thread.

[0039] In this embodiment, the popularity prediction score of all data objects collected by the system in the past 7 days is taken as the 80th percentile as the popularity threshold.

[0040] S4: Continuously monitor the current business I / O load of the data object to be migrated. When the current business I / O load is lower than the business load threshold, start the background migration thread, execute step S5, and determine the available migration throughput based on the business I / O bandwidth limit, the current business I / O load, the minimum business protection bandwidth, and the migration thread safety factor.

[0041] When the current business I / O load is lower than the business load threshold, the background migration thread is started. The real-time migration throughput does not exceed the available migration throughput and is reduced or paused when the business I / O load increases.

[0042] Available migration throughput is the non-negative throughput after reducing the upper limit of service I / O bandwidth by the migration thread safety factor, and then subtracting the current service I / O load and minimum service protection bandwidth. The calculation formula is: in, Available migration throughput; To improve thread safety during migration; This is the upper limit of service I / O bandwidth; This represents the current business I / O load. Minimum service protection bandwidth.

[0043] S5: Based on expected communication efficiency, overall topology vulnerability index, pre-migration candidate set and available migration throughput, determine the migration object, migration path and migration window.

[0044] For each data object and candidate migration path in the pre-migration candidate set, a migration priority score is calculated. The migration priority score increases with the increase of the heat prediction score, media adaptation benefits and expected communication efficiency, and decreases with the increase of the overall topology vulnerability index and the current service I / O load.

[0045] The formula for calculating migration priority score is: in, Score the priority of data object o as it migrates via candidate path r; The predicted popularity score for data object o; The media adaptation benefit for data object o is the improvement in the matching degree of storage media (source storage node and target storage node) before and after the data object migration; The expected communication efficiency of candidate migration path r; This is the overall topological vulnerability index; This is the penalty coefficient for business load. This is the normalized value of the current service I / O load. , This represents the historical peak of the business I / O load. Data objects with the highest priority scores are selected sequentially from highest to lowest, along with their corresponding migration paths, until the total data volume of the selected data objects reaches the upper limit of the available migration throughput within the current migration window. This determines the migration objects and their corresponding migration paths.

[0046] The migration window is determined by combining the available migration throughput and the preset migration throughput cap.

[0047] S6: Divide the migration object into multiple migration units based on the available migration throughput and migration window, and establish a flexible consistent snapshot layer based on copy-on-write snapshots for each migration unit.

[0048] The size of the migration unit is dynamically determined based on the length of the migration window and the available migration throughput. The formula for calculating the migration unit size is: in, Size of the migration unit; The maximum amount of data allowed for a single migration unit; Available migration throughput; This represents the current migration window length.

[0049] The migration object is divided into multiple migration units by continuous ranges of logical block addresses from the source volume, with each segment not exceeding the size of a migration unit. A unique identifier is assigned to each migration unit, and a mapping table is established between the migration unit, the source volume offset, and the target volume offset, serving as the addressing basis for subsequent data transfer.

[0050] The flexible consistency snapshot layer is a logical isolation layer established for each migration unit, consisting of three components: the source snapshot version, the migration view version, and the write-time replication difference record. The specific process includes: Source snapshot generation: Before the migration starts, the system performs a copy-on-write snapshot operation on the corresponding source volume of the source storage node to freeze the data state of the migration unit at the moment the migration begins.

[0051] Establishing a migration view version: The migration view is the data view seen by business I / O during the migration process. Its version number MigrViewVer is initialized to SrcSnapVer when the migration unit starts. Business read requests prioritize locating data based on MigrViewVer; when a business write hits a migration unit that is being migrated, the written data is recorded in the write-on-write difference record. MigrViewVer does not increment immediately but remains stable to ensure the repeatability of the migration process; after the migration unit is successfully committed, MigrViewVer switches to the latest committed version, and the source snapshot can be reclaimed.

[0052] Structure and maintenance of copy-on-write difference records: Copy-on-write difference records are write difference logs maintained by the source end for each migration unit. They are used to record the write operations of the business on the source volume after the migration begins, and adopt an append write structure similar to an LSM tree (Log-Structured Merge-Tree).

[0053] The flexibility of the flexible consistency snapshot layer is reflected in the following three dimensions: Version flexibility: The migration view version number is not forced to increment during migration, allowing copy-on-write differences on the source side to accumulate continuously without affecting the migrated data on the target side; Granularity flexibility: Each migration unit independently maintains the snapshot version and write-time replication difference log, refining the granularity to a single migration unit and avoiding the space overhead caused by full-volume snapshots; Rollback flexibility: When a migration unit fails, only the write-time replication difference record of that unit and the corresponding copy on the target end need to be discarded, without affecting other migration units that have been successfully migrated.

[0054] Before migration begins, the address range of the source volume logical blocks corresponding to each migration unit is pre-scanned during idle I / O cycles of the storage system to generate a pre-computed verification metadata index for each data block in the migration unit. Since this index is generated during idle I / O cycles before migration, the target data blocks can be directly verified against the index after the target end has finished writing, thereby reducing real-time full hash calculations during the migration process. The pre-computed verification metadata index can be represented as: in, For pre-computed verification metadata index; Used as a data block identifier; The hash value of the data block; For logical timestamps; For snapshot version identification; This is the source volume offset address; This is the identifier for the migration unit.

[0055] If the hash value, logical timestamp, or snapshot version identifier of the target data block is inconsistent with the index, the system will mark the corresponding migration unit as a candidate for verification failure, instead of immediately rolling back the entire migration object. This index also creates a secondary index based on the migration unit identifier for subsequent unit-level rollback and index compression.

[0056] S7: Based on migration units, migration paths, migration windows, and a flexible consistency snapshot layer, perform data migration of the data objects to be migrated.

[0057] This step employs a phased, recoverable, unit-level migration execution mechanism, sequentially executing five phases—read, transfer, write, verification, and commit—for each migration unit, while processing multiple migration units in parallel.

[0058] (1) Reading Phase: The migration thread reads the data blocks of the migration unit from the source storage node. The reading process is executed in batches at the data block granularity. The migration thread locates the data view to read through the migration view version corresponding to the migration unit: if there is an overriding record in the copy-on-write difference record of the data block, the copy-on-write difference record and the basic data of the source snapshot version are merged and returned to ensure that the read data is consistent with the consistency state at the moment the migration of the migration unit begins; if there is no overriding record in the data block, it is read directly from the data block corresponding to the source snapshot version. The write operation of the business during the migration does not block the reading phase. The written data is appended to the copy-on-write difference record of the migration unit, and the migration view version remains stable, thereby ensuring the repeatability and atomicity of the data obtained in the reading phase.

[0059] (2) Transmission phase: The read data blocks are transmitted to the target storage node via the migration path determined in step S5.

[0060] (3) Write phase: After receiving the data block, the target storage node writes it according to the target volume LBA. The target side maintains an independent write buffer for each migration unit, and does not cross-dependency with the write-time replication difference record of the same migration unit on the source side, ensuring that the granular flexibility of the flexible consistency snapshot layer also holds true on the write path. The write operations of different migration units are isolated from each other on the target side, and the write failure of a single migration unit does not pollute the target side data of other migration units.

[0061] (4) Verification Phase: After a migration unit is written on the target end, the data blocks corresponding to the migration unit are verified according to the pre-calculated verification metadata index. An incremental hash chain is generated for the data blocks in the migration unit that have changed between the snapshot creation time (the data state corresponding to SrcSnapVer) and the current verification time (including the data state after accumulated changes to the write-time replication difference record). The incremental hash chain is iteratively generated from the hash values ​​of the data blocks that have changed between adjacent snapshot versions, the hash value of the previous chain node, the logical timestamp, and the migration unit identifier. An incremental hash chain node can be represented as: in, This is the k-th incremental hash chain node; The hash value of the kth changed data block; This is the previous incremental hash chain node; This is the logical timestamp of the kth changed data block; This is the identifier of the migration unit to which the k-th changed data block belongs; For splicing operations; For hash functions, such as SHA-256 or MD5.

[0062] The verification phase leverages the version flexibility of the flexible consistency snapshot layer to include the incremental changes accumulated in the write-time replication difference record into the calculation scope of the incremental hash chain without modifying the data blocks that have been stably written to the target end. This allows the verification process to accurately capture all incremental changes between the snapshot creation time and the verification time without affecting the data already completed on the target end.

[0063] When the verification phase determines that there is a write conflict on the target end, inconsistent logical timestamps, or incremental hash chain verification failure, the rollback mechanism of the flexible consistency snapshot layer is triggered: only the incremental write of the corresponding failed migration unit is revoked, the write-time replication difference record of the migration unit and the corresponding copy on the target end are discarded, and the corresponding failed migration unit is added back to the migration queue. The migration path is recalculated based on the latest business I / O load, expected communication efficiency, and overall topology vulnerability index. Other migration units that have passed verification or are still in progress are unaffected, and their flexible consistency snapshot layer components (source snapshot version, migration view version, and write-time replication difference record) are retained normally. Migration units that have been successfully migrated remain valid. When the verification passes, the metadata pointer switch of the corresponding migration unit is committed.

[0064] (5) Commit phase: New metadata pointers are written independently to the source and target ends. After successful recording, the migration unit status is set to complete. The migration view version of the migration unit is switched to the latest committed version, its source snapshot version can be reclaimed, and its copy-on-write difference record is marked as cleanable.

[0065] Once all migration units for the same migration object have been committed, the system merges the pre-computed verification metadata index and incremental hash chain digest of the corresponding migration unit, and cleans up the committed copy-on-write difference records. If there are uncommitted or failed migration units, the system retains their verification metadata index and copy-on-write difference records until the migration is completed again or manual confirmation and cleanup are performed.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data migration method for a database, characterized in that, include: S1: Collect operational status data of the source storage node cluster, the target storage node cluster, and the network devices between them; S2: Construct a migration topology graph using storage nodes and network devices as nodes and the topological connections between storage nodes and network devices as edges; The running status data is input into the graph neural network model, and the aggregation features are obtained after the neighbor information is aggregated. The expected communication efficiency and overall topology vulnerability index of each candidate migration link are calculated based on the aggregation features. S3: Obtain the business access logs of the data objects to be migrated and perform online streaming time-series analysis. Determine the pre-migration candidate set based on access frequency, first-order growth of access frequency, recent access interval, and object business priority. S4: Continuously monitor the current business I / O load of the data object to be migrated. When the current business I / O load is lower than the business load threshold, execute step S5 and determine the available migration throughput based on the business I / O bandwidth limit, the current business I / O load, the minimum business protection bandwidth, and the migration thread safety factor. S5: Based on expected communication efficiency, overall topology vulnerability index, pre-migration candidate set and available migration throughput, determine migration objects, migration paths and migration windows; S6: Divide the migration object into multiple migration units based on the available migration throughput and migration window; S7: Perform data migration of the data objects to be migrated based on migration unit, migration path, and migration window.

2. The database data migration method as described in claim 1, characterized in that, In step S2, the degree of insufficient remaining space of nodes, the degree of link congestion, the risk of storage queue depth and the risk of historical link failure are extracted from the aggregated features, and the overall topology vulnerability index is calculated by weighted summation.

3. The database data migration method as described in claim 1, characterized in that, In step S3, online streaming time-series analysis is performed on the business access logs. Within a sliding time window, the access frequency, first-order growth rate of the access frequency, the most recent access interval, and the business priority of the object are statistically analyzed to calculate the popularity prediction score. The calculation formula is as follows: in, The predicted popularity score for data object o; This is the access frequency coefficient; The frequency of access to data object o; This is the coefficient for the increase in access frequency; The first-order increment of the access frequency of data object o; The penalty coefficient for the most recent access interval; The most recent access interval for data object o; To prevent small positive numbers from being divided by zero; This is the business priority coefficient; The business priority of data object o; When the predicted popularity score exceeds the popularity threshold, the corresponding data object is added to the pre-migration candidate set.

4. The database data migration method as described in claim 1, characterized in that, In step S4, the available migration throughput is the non-negative throughput after the upper limit of the service I / O bandwidth is reduced by the migration thread safety factor, and then the current service I / O load and the minimum service protection bandwidth are deducted.

5. A database data migration method as described in claim 3, characterized in that, In step S5, a migration priority score is calculated for each data object and candidate migration path in the pre-migration candidate set. The migration priority score increases with the increase of the heat prediction score, media adaptation benefits and expected communication efficiency, and decreases with the increase of the overall topology vulnerability index and the current service I / O load. Based on the migration priority score, the migration objects and their corresponding migration paths are determined; The migration window is determined by combining the available migration throughput and the preset migration throughput cap.

6. The database data migration method as described in claim 5, characterized in that, The formula for calculating migration priority score is: in, Score the priority of data object o as it migrates via candidate path r; Benefits of media adaptation for data object o; The expected communication efficiency of candidate migration path r; This is the overall topological vulnerability index; This is the penalty coefficient for business load. This is the normalized value of the current business I / O load.

7. The database data migration method as described in claim 1, characterized in that, In step S6, the size of the migration unit is dynamically determined based on the length of the migration window and the available migration throughput. The formula for calculating the migration unit size is as follows: in, Size of the migration unit; The maximum amount of data allowed for a single migration unit; Available migration throughput; This is the current migration window length; The migration object is divided into multiple migration units based on the size of the migration unit.

8. A database data migration method as described in claim 1, characterized in that, In step S6, a flexible consistent snapshot layer based on copy-on-write snapshots is established for each migration unit; In step S7, data migration of the data objects to be migrated is performed based on the flexible consistency snapshot layer; The flexible consistency snapshot layer consists of three components: source snapshot version, migration view version, and copy-on-write difference record. The source snapshot version is the data state of the migration unit frozen by performing a copy-on-write snapshot operation on the corresponding source volume of the source storage node before the migration starts. The migration view version is the version identifier of the data view seen by business I / O during the migration process. The copy-on-write difference record is used to record the write operations of the business on the source volume after the migration starts.

9. A database data migration method as described in claim 8, characterized in that, In step S7, when a verification failure occurs after the migration unit is written on the target end, the rollback flexible mechanism of the flexible consistency snapshot layer is triggered. Only the incremental write of the corresponding failed migration unit is canceled, the write-time replication difference record of the migration unit and the corresponding copy on the target end are discarded, and the corresponding failed migration unit is added back to the migration queue.

10. A database data migration method as described in claim 1, characterized in that, Before the migration starts, the source volume logical block address range corresponding to each migration unit is pre-scanned using the storage system's idle I / O cycle to generate a pre-computed verification metadata index for each data block of the migration unit. After the migration unit is written on the target side, the pre-computed verification metadata index is used to check the target side data block to reduce real-time full hash calculation during the migration process.

Images