Distributed database data migration method and device, equipment and medium
By acquiring parallel migration plans, real-time monitoring, and adaptive adjustments, problems such as uneven node load and network bottlenecks in distributed database migration were solved, improving migration efficiency and consistency, and achieving a highly efficient data migration process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-12
- Publication Date
- 2026-03-13
Smart Images
Figure CN121658463A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of big data technology and distributed technology, specifically to a distributed database data migration method, apparatus, device, medium, and program product. Background Technology
[0002] In the current big data platform environment, data migration often involves petabyte-scale data volumes, especially for distributed databases based on massively parallel processing. Their data is distributed across multiple nodes, presenting multiple challenges during migration: To avoid impacting business operations, existing cluster migrations often employ offline shutdown methods; data verification is frequently performed offline, either fully or by sampling, after migration is complete, consuming business downtime. Furthermore, existing solutions lack global allocation and parallel migration control for source data shards to target nodes during migration planning and execution, easily leading to uneven node loads, network bottlenecks, and low migration efficiency; consistency verification is delayed, requiring post-migration location and repair for inconsistencies, duplications, or data loss during migration; anomaly recovery mechanisms are inadequate, often relying on manual handling for task interruptions or node failures; and insufficient monitoring and scheduling linkage makes it difficult to promptly obtain and utilize performance metrics to adaptively adjust migration strategies. Summary of the Invention
[0003] In view of the above problems, this application provides a distributed database data migration method, apparatus, device, medium and program product.
[0004] According to a first aspect of this application, a distributed database data migration method is provided, comprising: obtaining a data migration plan, the data migration plan including: allocation information of each source data shard in the source distributed database cluster being assigned to a target node in the target distributed database cluster, and a parallel migration degree representing the number of source data shards to be migrated concurrently; executing the data migration plan, including: controlling the parallel migration of m source data shards, where m is a positive integer and m is not greater than the parallel migration degree; and migrating each source data shard to the assigned target node based on the allocation information, wherein, after the migration is completed, a target data shard corresponding to the source data shard is obtained at the target node.
[0005] According to an embodiment of this application, the distributed database data migration method further includes obtaining execution performance indicators by monitoring the execution process of the data migration plan; adjusting the data migration plan when the execution performance indicators meet the adaptive adjustment triggering conditions; wherein, executing the data migration plan further includes: executing the adjusted data migration plan in response to the adjustment of the data migration plan.
[0006] According to an embodiment of this application, the data migration plan further includes the data packet size and transmission rate used when transmitting source data fragments, and the performance indicators include data transmission latency and packet loss rate; wherein, when the performance indicators meet the adaptive adjustment triggering conditions, adjusting the data migration plan includes: when the duration for which the latency is greater than a first latency threshold and the packet loss rate is greater than a preset packet loss rate threshold reaches a first duration threshold, obtaining the ratio of the first latency threshold to the current latency; using the ratio as an adjustment coefficient to reduce the data packet size or reduce the transmission rate.
[0007] According to an embodiment of this application, after reducing the data packet size or lowering the transmission rate, the distributed database data migration method further includes: when the duration of the delay being less than a second delay threshold reaches a second duration threshold, restoring the reduced data packet size or the reduced transmission rate to the value before adjustment.
[0008] According to an embodiment of this application, the execution performance metrics include the load performance metrics of the first target node. When the execution performance metrics meet the adaptive adjustment triggering conditions, the data migration plan adjustment includes: when the load performance metrics of the first target node exceed the load threshold, finding a second target node whose load performance metrics are lower than the load threshold; and reallocating at least a portion of the source data shards allocated to the first target node in the allocation information to the second target node.
[0009] According to an embodiment of this application, obtaining a data migration plan includes: collecting basic migration information of the source distributed database cluster and the target distributed database cluster; and obtaining a data migration plan based on the basic migration information.
[0010] According to an embodiment of this application, migrating each source data shard to the allocated target node based on allocation information includes: determining the initial data state of the source data shard at the start of the migration; migrating the source data shard to the target node based on the initial data state to obtain the target data shard; and performing a consistency check on the target data shard on the target node after the migration is completed.
[0011] According to an embodiment of this application, the consistency verification of the target data shard on the target node includes: calculating the first hash value of the source data shard in the initial data state; calculating the second hash value of the target data shard; comparing the first hash value and the second hash value; when the first hash value and the second hash value are consistent, the target data shard verification is determined to be successful; when the first hash value and the second hash value are inconsistent, the target data shard verification is determined to be unsuccessful, and an exception handling process is triggered.
[0012] According to an embodiment of this application, the exception handling process includes: identifying the exception type based on the result information of the verification failure; obtaining the repair strategy corresponding to the exception type from the preset exception repair rule base; executing the repair strategy to repair the target data shard; and performing consistency verification again on the repaired target data shard.
[0013] According to an embodiment of this application, the distributed database data migration method further includes: monitoring data changes in the source distributed database cluster during the execution of the data migration plan and forming an incremental data change log; after all source data shards in the source distributed database cluster have been migrated and the consistency verification has passed, applying the data changes to the corresponding nodes of the target distributed database cluster according to the incremental data change log.
[0014] According to an embodiment of this application, the distributed database data migration method further includes: during the application of data changes corresponding to incremental data change logs to the target distributed database cluster, continuously acquiring subsequent data changes generated by the source distributed database cluster, and determining the data volume of subsequent data changes to be applied; when the data volume is greater than a preset threshold, applying subsequent data changes to the corresponding node of the target distributed database cluster, and continuously acquiring subsequent data changes generated by the source distributed database cluster during the application period, re-determining the data volume after the application is completed, and repeating the above application operation; when the data volume is reduced to or below the preset threshold, applying subsequent data changes to the corresponding node of the target distributed database cluster, and connecting business traffic to the target distributed database cluster during the application process.
[0015] Another aspect of this application provides a distributed database data migration apparatus, comprising: a data migration plan acquisition module for acquiring a data migration plan, the data migration plan including: allocation information of each source data shard in the source distributed database cluster being assigned to a target node in the target distributed database cluster, and a parallel migration degree representing the number of source data shards being migrated concurrently; and a data migration plan execution module for executing the data migration plan, including: controlling the parallel migration of m source data shards, where m is a positive integer and m is not greater than the parallel migration degree; and migrating each source data shard to the assigned target node based on the allocation information, wherein, after the migration is completed, a target data shard corresponding to the source data shard is obtained at the target node.
[0016] Another aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.
[0017] Another aspect of this application provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.
[0018] Another aspect of this application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.
[0019] The embodiments of this application obtain a data migration plan that includes the allocation information from source data shards to target nodes and the degree of parallel migration, and control the parallel migration of source data shards during the execution phase. This enables the migration process to achieve controllable concurrency and targeted landing point migration at the shard granularity, and forms target data shards corresponding to the source data shards on the target nodes. This improves the organizational efficiency and resource utilization of large-scale distributed database migration and reduces the risk of resource contention caused by disordered concurrency. Attached Figure Description
[0020] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:
[0021] Figure 1 The illustration shows an application scenario diagram of a distributed database data migration method, apparatus, device, medium, and program product according to embodiments of this application;
[0022] Figure 2 A flowchart illustrating a distributed database data migration method according to an embodiment of this application is shown schematically.
[0023] Figure 3 A flowchart illustrating a data migration plan according to an embodiment of this application is shown schematically.
[0024] Figure 4 This illustration shows a flowchart of migrating each source data shard to the assigned target node based on allocation information, according to an embodiment of this application.
[0025] Figure 5 This illustration schematically shows a flowchart of a consistency check for target data fragments on a target node according to an embodiment of this application;
[0026] Figure 6 A flowchart illustrating the adaptive adjustment of a data migration plan according to an embodiment of this application is shown schematically.
[0027] Figure 7 This illustration schematically shows one of the flowcharts for adjusting a data migration plan based on performance metrics according to an embodiment of this application;
[0028] Figure 8This illustration schematically shows a second flowchart of adjusting a data migration plan based on performance metrics according to an embodiment of this application;
[0029] Figure 9 A flowchart illustrating an exception handling process according to an embodiment of this application is shown schematically.
[0030] Figure 10 One of the flowcharts for incremental data migration according to an embodiment of this application is illustrated schematically;
[0031] Figure 11 A second flowchart illustrating incremental data migration according to an embodiment of this application is shown schematically.
[0032] Figure 12 This schematically illustrates a structural block diagram of a distributed database data migration apparatus according to an embodiment of this application; and
[0033] Figure 13 A block diagram schematically illustrates an electronic device suitable for implementing a distributed database data migration method according to an embodiment of this application. Detailed Implementation
[0034] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.
[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0036] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0037] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0038] Figure 1 The illustration shows an application scenario diagram of a distributed database data migration method, apparatus, device, medium, and program product according to embodiments of this application.
[0039] like Figure 1 As shown, application scenario 100 according to this embodiment may include source database cluster nodes 101 and 102, target database cluster nodes 106 and 107, network 104, and migration control server 105. Network 104 serves as a medium for providing communication links between source database cluster nodes 101 and 102, target database cluster nodes 106 and 107, and migration control server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0040] The migration control server 105 can interact with the source database cluster nodes 101 and 102 and the target database cluster nodes 106 and 107 via network 104 to collect basic migration information, issue migration plans, and monitor migration progress. The source database cluster nodes 101 and 102 store the source data shards to be migrated, while the target database cluster nodes 106 and 107 are used to receive the migrated data and form target data shards.
[0041] The source database cluster nodes 101 and 102 and the target database cluster nodes 106 and 107 can be various types of database servers, including but not limited to distributed database nodes based on massively parallel processing architectures, relational database servers, etc.
[0042] The migration control server 105 can be a server that provides data migration control and scheduling services, such as performing functions like formulating data migration plans, scheduling and allocating migration tasks, and monitoring and adaptively adjusting the migration process (this is just an example). The migration control server 105 can analyze and process data such as the collected status information of the source and target clusters and performance indicators during the migration execution process, and adjust the migration strategy based on the analysis results.
[0043] It should be noted that the distributed database data migration method provided in this application embodiment can generally be executed by the migration control server 105. Correspondingly, the distributed database data migration device provided in this application embodiment can generally be located in the migration control server 105. The distributed database data migration method provided in this application embodiment can also be executed by a server or server cluster that is different from the migration control server 105 and capable of communicating with the source database cluster nodes 101, 102 and / or the target database cluster nodes 106, 107. Correspondingly, the distributed database data migration device provided in this application embodiment can also be located in a server or server cluster that is different from the migration control server 105 and capable of communicating with the source database cluster nodes 101, 102 and / or the target database cluster nodes 106, 107.
[0044] It should be understood that Figure 1 The number of source database cluster nodes, target database cluster nodes, network, and migration control servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of source database cluster nodes, target database cluster nodes, network, and migration control servers can be used.
[0045] The following will be based on Figure 1 The described scene, through Figures 2-11 A distributed database data migration method according to embodiments of this application will be described in detail.
[0046] Figure 2 A flowchart illustrating a distributed database data migration method according to an embodiment of this application is shown.
[0047] like Figure 2 As shown, the distributed database data migration method of this embodiment includes operations S210 to S220.
[0048] In operation S210, a data migration plan is obtained, which includes: allocation information of each source data shard in the source distributed database cluster to the target node of the target distributed database cluster, and parallel migration degree representing the number of source data shards to be migrated concurrently.
[0049] Distributed databases distribute data across multiple nodes, achieving high-performance computing through parallel processing, such as distributed databases based on Massively Parallel Processing (MPP). In this embodiment, the source and target distributed database clusters need to be analyzed to formulate a data migration plan. The data migration plan includes two core elements: allocation information and parallel migration degree. Allocation information defines the mapping relationship between source data shards and target nodes, and its rationality directly affects the query performance of the target cluster. Parallel migration degree limits the maximum number of source data shards that can be migrated simultaneously. It requires comprehensive consideration of factors such as the processing capacity, network bandwidth, and storage read / write performance of both the source and target clusters to ensure migration efficiency while avoiding cluster overload.
[0050] To explain in more detail how to obtain a data migration plan, the following will combine... Figure 3 To elaborate further.
[0051] Figure 3 A flowchart illustrating a data migration plan according to an embodiment of this application is shown.
[0052] like Figure 3 As shown, the data migration plan in this embodiment includes operations S211 to S212.
[0053] During operation S211, basic migration information for the source and target distributed database clusters is collected.
[0054] In this embodiment, the scope of data collection for migration includes: cluster node topology, i.e., the distribution of each node and the connection relationships between nodes; node hardware configuration, such as processor parameters, memory capacity, disk capacity and type; data distribution, including the number, size and distribution rules of data shards stored on each node; real-time node load status, such as performance indicators like processor utilization, memory utilization, and disk I / O (Input / Output, read / write) utilization; and network topology information, including internal cluster bandwidth and inter-node network latency. For the target cluster, it is also necessary to assess whether the available storage space is sufficient to accommodate the migration data.
[0055] In operation S212, a data migration plan is obtained based on the migration foundation information.
[0056] In the embodiments of this application, a data migration plan is generated by comprehensively utilizing scheduling algorithms based on the collected migration baseline information. First, an allocation strategy is determined according to the number of nodes and hardware configuration of the source and target clusters. For example, when the number of nodes in the target cluster is n times that of the source cluster, data fragments from each node in the source cluster are allocated to n corresponding nodes in the target cluster to utilize the parallel processing capabilities of the target cluster. Second, a reasonable parallel migration degree is determined based on network bandwidth, storage I / O capacity, and current service load. The parallel migration degree needs to balance migration efficiency and stability: too high a degree may lead to network congestion or node overload, affecting online services; too low a degree will prolong migration time. Furthermore, the data migration plan may also include parameters such as packet size and transmission rate, the initial values of which can be determined based on historical information.
[0057] Once the data migration plan is obtained, execute the plan to complete the actual data migration work.
[0058] In operation S220, a data migration plan is executed, including: controlling the parallel migration of m source data shards, where m is a positive integer and m is not greater than the parallel migration degree; and migrating each source data shard to the assigned target node based on the allocation information, wherein, after the migration is completed, the target data shard corresponding to the source data shard is obtained at the target node.
[0059] In the embodiments of this application, when performing migration operations according to the data migration plan, a multi-threaded, node-level parallelization strategy is adopted to migrate multiple source data shards simultaneously. m data shards are selected from the queue of source data shards to be migrated and migration is started simultaneously, transmitting the source data shard content to the corresponding target node. After receiving the source data shard content, the target node creates the corresponding target data shard in its local storage. By controlling the concurrency through parallel migration, migration efficiency can be improved while avoiding overload, ensuring the operation of online services.
[0060] To ensure the accuracy and consistency of data migration, the following will be combined with... Figure 4 This explains how to migrate each source data shard to the assigned target node based on the allocation information.
[0061] Figure 4 The illustration shows a flowchart of migrating each source data shard to the assigned target node based on allocation information, according to an embodiment of this application.
[0062] like Figure 4 As shown, the migration of each source data shard to the allocated target node based on the allocation information in this embodiment includes operations S221 to S223.
[0063] In operation S221, at the start of the migration, the initial data state of the source data shards is determined.
[0064] In operation S222, based on the initial data state, the source data shards are migrated to the target node to obtain the target data shards.
[0065] In the embodiments of this application, since the distributed database continuously receives business requests and generates data changes during operation, if the initial data state of the source data shard is not fixed, the content of the source data shard may change during data transmission, resulting in the data transmitted to the target node being in an uncertain intermediate state, making consistency verification impossible. The initial data state can be determined using methods such as database snapshot technology, creating a point-in-time snapshot for the source data shard to be migrated, recording the complete data state of the data shard at a specific moment. Subsequent data transmission operations are based on this record, transmitting fixed data rather than real-time data. Even if the source data shard data changes during data transmission, it will not affect the data content being transmitted, ensuring that the data transmitted to the target node is consistent with the initial data state of the source data shard. After receiving the transmitted data content, the target node writes it to its local storage to form the target data shard.
[0066] During operation S223, after the migration is completed, a consistency check is performed on the target data shards on the target node.
[0067] In the embodiments of this application, after the source data fragment content is transmitted to the target node and forms the target data fragment, a consistency check is performed on the target data fragment to verify the effect of the data migration, confirming that the content of the target data fragment is consistent with the content of the source data fragment in its initial data state, and that no data loss, deviation, or corruption has occurred. This real-time verification mechanism of migrating and checking simultaneously can detect problems earlier than traditional offline verification, and can shorten the overall migration time. The consistency check can use lightweight methods such as hash value comparison, reducing performance overhead and improving efficiency.
[0068] The following is combined Figure 5 Explain how to perform consistency checks on target data shards.
[0069] Figure 5 The flowchart illustrating the consistency verification of target data fragments on a target node according to an embodiment of this application is shown in the illustration.
[0070] like Figure 5 As shown, the consistency verification of target data fragments on the target node in this embodiment includes operations S2231 to S2234.
[0071] In operation S2231, the first hash value of the source data fragment in the initial data state is calculated.
[0072] In operation S2232, the second hash value of the target data fragment is calculated.
[0073] In operation S2233, the first hash value and the second hash value are compared.
[0074] In operation S2234, when the first hash value and the second hash value are consistent, the target data fragment verification is determined to be successful; when the first hash value and the second hash value are inconsistent, the target data fragment verification is determined to be unsuccessful, and the exception handling process is triggered.
[0075] A hash calculation is performed on the data content of the source data fragment in the initial data state determined in operation S221 to obtain a first hash value. This first hash value serves as a unique identifier for the content of the source data fragment in the initial data state and is used for subsequent comparison and verification. The same hash calculation is performed on the target data fragment content already stored on the target node to obtain a second hash value. The target data fragment may be corrupted or lost during transmission, which will be reflected in changes to the hash value. A bit-by-bit comparison between the first and second hash values can determine whether the two data fragment contents are consistent. Each node and each data fragment can be verified independently and in parallel, without waiting for all data transmission to complete, thus achieving a streaming comparison mechanism of verification while migrating. If the hash values match, the target data fragment verification is deemed successful, the data migration is successful, and the data is error-free; if the hash values do not match, the target data fragment verification is deemed unsuccessful, triggering an exception handling process.
[0076] During data migration, network conditions and node load may fluctuate. This application provides an adaptive adjustment mechanism to dynamically optimize the migration strategy based on real-time performance indicators during the migration process.
[0077] Figure 6 A flowchart illustrating adaptive adjustment of a data migration plan according to an embodiment of this application is shown schematically.
[0078] like Figure 6 As shown, the adaptive adjustment of the data migration plan includes operations S310 to S320.
[0079] By operating S310, performance metrics are obtained by monitoring the execution process of the data migration plan.
[0080] In the embodiments of this application, the execution status of the migration task is continuously monitored in real time throughout the entire process of executing the data migration plan, and various performance indicator data are collected. Execution performance indicators include multiple dimensions: data transmission related indicators, such as network latency, packet loss rate, transmission rate, and bandwidth utilization; node load related indicators, such as processor utilization, memory utilization, and disk I / O utilization of each target node; and migration task progress related indicators, such as the number of data shards that have been migrated, the number of remaining data shards to be migrated, and the estimated completion time.
[0081] When operating S320, adjust the data migration plan when the performance metrics meet the adaptive adjustment trigger conditions.
[0082] In the embodiments of this application, based on the execution performance metrics obtained in operation S310, it is determined whether the current data migration plan execution status meets the preset adaptive adjustment trigger conditions. The adaptive adjustment trigger conditions can be set with different thresholds and judgment logic according to different performance metric types. When it is detected that an execution performance metric meets a certain adaptive adjustment trigger condition, the data migration plan will be modified accordingly according to the preset adjustment strategy. In one embodiment, the execution performance metrics can also be converted into real-time alarms and visualizations to remind operations personnel to make adjustments. In another embodiment, a large model can be introduced for decision-making, determining the adjustment direction of the data migration plan based on the execution performance metrics.
[0083] Furthermore, the data migration plan execution method in the distributed database data migration method also includes: in response to adjustments to the data migration plan, executing the adjusted data migration plan.
[0084] In this embodiment, after the data migration plan is adjusted in operation S320, the adjusted plan needs to be applied to the ongoing migration task. Specifically, if the adjustment involves a change in the degree of parallel migration, the number of migration tasks currently being executed in parallel needs to be changed accordingly; if the adjustment involves a change in transmission parameters, the tasks currently transmitting need to continue transmitting using a new data packet size or transmission rate; if the adjustment involves a change in the source data fragment allocation scheme, the source data fragments that have not yet started migrating need to be rescheduled to the newly allocated target nodes.
[0085] To illustrate how the adaptive adjustment mechanism is implemented, two embodiments based on network transmission performance indicators and node load performance indicators are described below.
[0086] Figure 7 One of the flowcharts illustrating the adjustment of a data migration plan based on performance metrics according to an embodiment of this application is shown schematically.
[0087] like Figure 7 As shown, the distributed database data migration method of this embodiment includes operations S410 to S420.
[0088] In this embodiment, the data migration plan further includes the packet size and transmission rate used when transmitting source data fragments, and the performance metrics include data transmission latency and packet loss rate; wherein, when the performance metrics meet the adaptive adjustment triggering conditions, adjusting the data migration plan includes:
[0089] In operation S410, when the duration for which the delay is greater than the first delay threshold and the packet loss rate is greater than the preset packet loss rate threshold reaches the first duration threshold, the ratio of the first delay threshold to the current delay is obtained.
[0090] In the embodiments of this application, an adaptive bandwidth algorithm is introduced to dynamically adjust the batch size and transmission rate of transmitted data according to the network status. During the migration process, network conditions may fluctuate due to network congestion, interference from other service traffic, etc. When network conditions deteriorate, leading to increased latency and packet loss rate, transmitting according to the original data packet size and transmission rate will reduce transmission efficiency or even cause transmission failure. The data migration plan pre-sets initial values for data packet size and transmission rate, which can be determined based on the network bandwidth and historical information between the source cluster and the target cluster. When the data transmission latency and packet loss rate are detected to exceed their respective thresholds simultaneously and persist for a certain period of time, it indicates that the current network conditions have deteriorated and are not sporadic fluctuations. The first latency threshold and the preset packet loss rate threshold are set according to the network baseline performance. For example, the first latency threshold can be set to 1.5 times the baseline latency, and the preset packet loss rate threshold can be set to 0.1%. The first duration threshold is used to filter out brief network jitters. Adjustments are only required when the duration of the latency and packet loss rate exceeding the standard exceeds the first duration threshold. At this point, the ratio of the first latency threshold to the current actual latency is calculated. This ratio reflects the degree of degradation of the current latency relative to the baseline latency. For example, if the first latency threshold is 50 ms and the current actual latency is 80 ms, then the ratio is 50 / 80 = 0.625.
[0091] When operating the S420, the ratio is used as an adjustment factor to reduce the data packet size or decrease the transmission rate.
[0092] The ratio calculated in operation S410 is used as an adjustment factor to reduce the packet size or transmission rate. The new packet size can be calculated using the formula "New packet size = Current packet size × Adjustment factor", and the new transmission rate can be calculated using the formula "New transmission rate = Current transmission rate × Adjustment factor". For example, if the adjustment factor is 0.625 and the current packet size is 1 MB, the new packet size will be adjusted to 1 MB × 0.625 = 0.625 MB; if the current transmission rate is 100 Mbps, the new transmission rate will be adjusted to 100 Mbps × 0.625 = 62.5 Mbps. By reducing the packet size, the transmission time of packets in the network and the probability of being affected by network fluctuations can be reduced; by reducing the transmission rate, the network load can be reduced, and network congestion can be alleviated.
[0093] After reducing the data packet size or lowering the transmission rate, the distributed database data migration method also includes: when the duration of the delay being less than the second delay threshold reaches the second duration threshold, restoring the reduced data packet or the lowered transmission rate to its pre-adjustment state.
[0094] In the embodiments of this application, after adjusting the packet size or transmission rate, changes in network performance indicators are continuously monitored. If the network condition improves, such as when the data transmission latency drops below a second latency threshold, and this improvement lasts for the time specified by a second duration threshold, the network is considered to have recovered, and the transmission parameters can be restored to their pre-adjustment values to improve transmission efficiency. The second latency threshold is typically set to a level close to the network baseline latency; the second duration threshold is used to confirm the stability of the network condition.
[0095] In addition to adjustments based on network transmission performance, the system can also be adjusted based on node load performance. The following section combines... Figure 8 Please provide an explanation.
[0096] Figure 8 The illustration shows a second flowchart of adjusting a data migration plan based on performance metrics according to an embodiment of this application.
[0097] like Figure 8 As shown, the distributed database data migration method of this embodiment includes operations S510 to S520.
[0098] In embodiments of this application, the performance metrics include the load performance metrics of the first target node, wherein adjusting the data migration plan when the performance metrics meet the adaptive adjustment triggering conditions includes:
[0099] In operation S510, when the load performance index of the first target node exceeds the load threshold, a second target node with a load performance index lower than the load threshold is searched.
[0100] In operation S520, at least a portion of the source data fragments allocated to the first target node in the allocation information are reallocated to the second target node.
[0101] In the embodiments of this application, the load on each target node may be uneven during the data migration process. Some nodes may bear a large number of migration tasks or process a large number of online business requests simultaneously, resulting in excessive load, manifested as processor utilization, memory utilization, disk I / O utilization, and other load performance indicators exceeding normal ranges. Continuing to allocate migration tasks to high-load nodes may lead to performance degradation or even failure of those nodes, affecting online business operations and data migration. By monitoring the load performance indicators of each target node, when the load performance indicator of a target node (referred to as the first target node) exceeds a preset load threshold, some or all of the source data shards originally allocated to that node need to be transferred to other nodes with lower loads. The load performance indicator can be a single indicator such as processor utilization, or a comprehensive evaluation value of multiple indicators. The load threshold setting needs to be determined based on the node's hardware configuration, for example, the load threshold for processor utilization can be set to 80%, and the load threshold for disk I / O utilization can be set to 70%, etc. Iterate through the other target nodes in the target distributed database cluster to find nodes with load performance metrics below the load threshold (referred to as second target nodes), which have sufficient remaining processing capacity to handle additional migration tasks. Modify the allocation information in the data migration plan, reallocating at least a portion of the source data shards originally allocated to the first target node to the second target node. The number of reassigned source data shards can be determined based on the degree of overload on the first target node and the remaining processing capacity of the second target node.
[0102] As described in operation S2234, an exception handling process is triggered when the consistency check fails. The following section combines... Figure 9 Please explain the process in detail.
[0103] Figure 9 A flowchart illustrating an exception handling process according to an embodiment of this application is shown.
[0104] like Figure 9 As shown, the exception handling process in this embodiment includes operations S610 to S640.
[0105] When operating S610, the anomaly type is identified based on the result information of the verification failure.
[0106] In the embodiments of this application, when the target data shard verification fails in operation S2234, the system triggers an exception handling process. The result information of the failed verification includes, for example, the hash values of the source and target data shards, the identifier of the migration task, information about the source and target nodes, transmission logs, and error codes. By analyzing this result information, the cause of data inconsistency can be determined. Common exception types include: network transmission anomalies, such as network interruptions or packet loss leading to incomplete data; node failures, such as hardware or software failures occurring in the source or target node during migration; and transaction conflicts, such as the source data shard being affected by concurrent transactions during migration. The exception type can be automatically identified through techniques such as keyword matching, error code mapping, and log pattern recognition.
[0107] When operating S620, the repair strategy corresponding to the exception type is obtained from the preset exception repair rule base.
[0108] When operating S630, a repair strategy is executed to repair the target data fragments.
[0109] Based on the anomaly type identified during S610 operation, the corresponding repair strategy is retrieved from the pre-built anomaly repair rule base. The anomaly repair rule base is a pre-built knowledge base that stores the mapping relationship between various anomaly types and corresponding repair strategies. Its structure includes fields such as anomaly type identifier, matching rules (e.g., regular expressions for error messages), repair actions (specific repair operation steps or scripts), and retry strategies (e.g., number of retries, retry interval). After querying by anomaly type identifier or matching rules to find the most matching repair strategy, the defined repair actions are executed to repair the target data fragments that failed verification. Repair actions include: full retransmission, which retransmits the source data fragments from the source node to the target node, overwriting incorrect target data fragments on the target node; differential retransmission, which compares the source and target data fragments to identify inconsistencies and only transmits the differential data to repair the target data fragment, significantly saving transmission time and bandwidth compared to full retransmission; and node switching, which, if the target node itself is faulty, reassigns the source data fragment to another target node for migration. For example, for the "network transmission interruption" exception type, the corresponding repair strategy might be "complete retransmission, retries 3 times"; for the "data hash value mismatch" exception type, the corresponding repair strategy might be "first attempt differential retransmission, if the verification still fails after retransmission, then perform a complete retransmission". Exception handling operations can be recorded in the audit log for subsequent investigation and tracing.
[0110] During operation of S640, a consistency check is performed again on the repaired target data fragments.
[0111] In the embodiments of this application, after the repair operation is completed, a consistency check needs to be performed again on the repaired target data shard to verify the repair effect. The consistency check process performed again is the same as the process described in operation S223. If the check passes again, it indicates that the repair operation is successful and the migration task of the source data shard is completed. If the check still fails again, the retry policy in the repair strategy can be used to decide whether to continue the repair attempt. For example, if the retry policy stipulates that three retries are allowed, and the first repair has failed, operation S630 will be executed again for a second repair; if the maximum number of retries has been reached and the check still fails, maintenance personnel can intervene to handle the situation.
[0112] Because the data migration process takes a long time, the source cluster is still processing business requests and generating data changes during this period. Therefore, it is necessary to synchronize these changes through an incremental data migration mechanism.
[0113] Figure 10 One of the flowcharts illustrating incremental data migration according to an embodiment of this application is shown schematically.
[0114] like Figure 10 As shown, the incremental data migration process in this embodiment includes operations S710 to S720.
[0115] When operating an S710, during the execution of a data migration plan, monitor data changes in the source distributed database cluster and generate incremental data change logs.
[0116] When operating the S720, after all source data shards in the source distributed database cluster have been migrated and consistency verification has passed, the data changes are applied to the corresponding nodes in the target distributed database cluster according to the incremental data change log.
[0117] In the embodiments of this application, during the execution of the data migration plan, data changes occurring in the source distributed database cluster are continuously monitored and recorded. Since the data migration process typically lasts a considerable amount of time, the source cluster continues to receive and process business requests during this period, constantly generating new data insertion, update, and deletion operations. If only the initial data state of the source data shards at the start of the migration is migrated, while ignoring the data changes generated during the migration process, the data in the target cluster will differ from that of the source cluster after the migration is completed. A CDC (Change Data Capture) mechanism is used to capture data changes in the source cluster in real time. This mechanism can be implemented through various techniques such as parsing the database transaction log, using database triggers, or intercepting changes at the application layer. The captured data change operations are recorded in the incremental data change log, which contains detailed information such as the type of change operation (insert, update, delete), the tables and data shards involved, the data values before and after the change, and the timestamp of the change. Once all source data shards have been migrated and passed consistency verification, the incremental data change log is read, and these change operations are applied sequentially to the corresponding nodes of the target cluster according to the order in which the data changes occurred. For each data change operation recorded in the log, firstly, based on the table and data shard information involved in the change, determine which node and which target data shard in the target cluster the change should be applied to; then, perform the corresponding data operation on the target node. For example, for an insert operation, insert a new record in the target data shard; for an update operation, update the value of the corresponding record in the target data shard; and for a delete operation, delete the corresponding record in the target data shard.
[0118] Since the source cluster will still generate new data changes during the application's incremental data changes, data consistency issues will arise. The following section will discuss this further. Figure 11 Explain the mechanism for multiple rounds of incremental data migration and business traffic switching.
[0119] Figure 11 A second flowchart illustrating incremental data migration according to an embodiment of this application is shown.
[0120] like Figure 11 As shown, the incremental data migration process in this embodiment also includes operations S810 to S830.
[0121] When operating S810, during the process of applying the data changes corresponding to the incremental data change log to the target distributed database cluster, the subsequent data changes generated by the source distributed database cluster are continuously acquired, and the amount of subsequent data changes to be applied is determined.
[0122] In the embodiments of this application, during the process of applying incremental data changes in operation S720, new data changes generated by the source cluster are continuously captured through the CDC mechanism. These new data changes are referred to as subsequent data changes. Since incremental data application takes time, and the source cluster continues to receive business requests, subsequent data changes will inevitably occur while the first batch of incremental data changes is being applied. The amount of data in these subsequent data changes is statistically analyzed in real time. The amount of data can be represented by the number of change operations, the size of the data involved, or other suitable metrics.
[0123] When operating S820, if the data volume exceeds a preset threshold, subsequent data changes are applied to the corresponding nodes of the target distributed database cluster. During the application process, subsequent data changes generated by the source distributed database cluster are continuously acquired. After the application is completed, the data volume is re-determined and the above application operation is repeated.
[0124] In the embodiments of this application, it is determined whether the amount of subsequent data changes counted in operation S810 exceeds a preset threshold. The preset threshold needs to be determined comprehensively based on factors such as the task's tolerance for data latency and the processing capacity of the target cluster. For example, it can be set to 10,000 change operations or 100 MB of data. If the amount of subsequent data changes exceeds the preset threshold, it indicates that the current data gap is large, and incremental data migration needs to continue. These subsequent data changes are applied to the corresponding nodes of the target cluster, and the application method is the same as described in operation S720. During the application of this batch of subsequent data changes, the updated subsequent data changes generated by the source cluster are continuously obtained through the CDC mechanism. After this batch of subsequent data changes is applied, the amount of newly generated subsequent data changes is recounted, and it is determined again whether it exceeds the preset threshold. If it still exceeds the preset threshold, the application operation is executed again, forming a loop process.
[0125] When operating S830, when the data volume decreases to or below a preset threshold, subsequent data changes are applied to the corresponding nodes of the target distributed database cluster, and business traffic is connected to the target distributed database cluster during the application process.
[0126] In the embodiments of this application, when the amount of subsequent data changes detected decreases to or below a preset threshold, it indicates that the data gap between the target cluster and the source cluster is within a controllable range, and the business traffic can be switched from the source cluster to the target cluster. Simultaneously with applying the last batch of subsequent data changes, business traffic is connected to the target cluster. In one instance, a dual-write strategy can be adopted, writing data to both the source and target clusters during the switchover period, and stopping writing to the source cluster after the target cluster has been running stably for a period of time. After the business traffic is connected to the target cluster, the target cluster directly receives and processes business requests. Since the target cluster has already started generating new data changes during the application of the last batch of incremental data, if the application of incremental data conflicts with the newly generated data changes in the target cluster, the new data from the target cluster takes precedence, ensuring the real-time performance and accuracy of the business data.
[0127] The distributed database data migration method provided in this application improves migration efficiency and resource utilization by formulating a data migration plan that includes allocation information and parallel migration degree, thereby achieving reasonable allocation of source data shards to target nodes and parallel migration execution. Combined with a complete process of consistency verification, adaptive adjustment, exception handling, incremental data migration, and business switching, it enables efficient and reliable online large-scale data migration, solving to some extent the problems of long downtime, low efficiency, delayed data consistency verification, and lack of self-healing capabilities inherent in traditional migration solutions.
[0128] Based on the above-described distributed database data migration method, this application also provides a distributed database data migration apparatus.
[0129] Figure 12 A schematic block diagram of a distributed database data migration apparatus according to an embodiment of this application is shown.
[0130] like Figure 12 As shown, the distributed database data migration device 900 of this embodiment includes a data migration plan acquisition module 910 and a data migration plan execution module 920.
[0131] The data migration plan acquisition module 910 is used to acquire a data migration plan, which includes: allocation information of each source data shard in the source distributed database cluster to the target node in the target distributed database cluster, and a parallel migration degree representing the number of source data shards being migrated concurrently. In one embodiment, the data migration plan acquisition module 910 can be used to perform the operation S210 described above, which will not be repeated here.
[0132] In some embodiments, the data migration plan acquisition module 910 is further used to collect basic migration information of the source distributed database cluster and the target distributed database cluster; and to obtain a data migration plan based on the basic migration information.
[0133] The data migration plan execution module 920 is used to execute a data migration plan, including: controlling the parallel migration of m source data shards, where m is a positive integer and m is not greater than the parallel migration degree; and migrating each source data shard to an assigned target node based on allocation information, wherein, after the migration is completed, a target data shard corresponding to the source data shard is obtained at the target node. In one embodiment, the data migration plan execution module 920 can be used to execute the operation S220 described above, which will not be repeated here.
[0134] In some embodiments, the data migration plan execution module 920 further includes a consistency verification submodule, which is used to determine the initial data state of the source data shard at the start of the migration; migrate the source data shard to the target node based on the initial data state to obtain the target data shard; and perform consistency verification on the target data shard on the target node after the migration is completed.
[0135] In some embodiments, the distributed database data migration apparatus further includes: a data migration plan adjustment module, used to obtain execution performance indicators by monitoring the execution process of the data migration plan; and to adjust the data migration plan when the execution performance indicators meet the adaptive adjustment trigger conditions; wherein, executing the data migration plan further includes: executing the adjusted data migration plan in response to the adjustment of the data migration plan. An incremental data module is used to monitor data changes in the source distributed database cluster during the execution of the data migration plan, forming an incremental data change log; and after all source data shards in the source distributed database cluster have been migrated and consistency verification has passed, to apply the data changes to the corresponding nodes of the target distributed database cluster according to the incremental data change log.
[0136] According to embodiments of this application, any plurality of modules in the data migration plan acquisition module 910 and the data migration plan execution module 920 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the data migration plan acquisition module 910 and the data migration plan execution module 920 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the data migration plan acquisition module 910 and the data migration plan execution module 920 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.
[0137] Figure 13 A block diagram schematically illustrates an electronic device suitable for implementing a distributed database data migration method according to an embodiment of this application.
[0138] like Figure 13 As shown, an electronic device 1000 according to an embodiment of this application includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.
[0139] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.
[0140] According to embodiments of this application, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the input / output (I / O) interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.
[0141] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.
[0142] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.
[0143] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the distributed database data migration method provided in the embodiments of this application.
[0144] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0145] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0146] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0147] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0149] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.
Claims
1. A distributed database data migration method, characterized in that, include: Obtain a data migration plan, which includes: allocation information of each source data shard in the source distributed database cluster to the target node of the target distributed database cluster, and a parallel migration degree representing the number of source data shards to be migrated concurrently; Executing the data migration plan includes: Control the parallel migration of m source data shards, where m is a positive integer and m is not greater than the parallel migration degree; and Based on the allocation information, each source data shard is migrated to the allocated target node, wherein, after the migration is completed, a target data shard corresponding to the source data shard is obtained at the target node.
2. The method according to claim 1, wherein, The method further includes: By monitoring the execution process of the data migration plan, performance metrics are obtained. When the execution performance metrics meet the adaptive adjustment trigger conditions, the data migration plan is adjusted; The execution of the data migration plan further includes: in response to an adjustment to the data migration plan, executing an adjusted data migration plan.
3. The method according to claim 2, wherein, The data migration plan also includes the packet size and transmission rate used when transmitting the source data fragments, and the performance metrics include data transmission latency and packet loss rate; wherein, adjusting the data migration plan when the performance metrics meet the adaptive adjustment trigger condition includes: When the duration for which the delay is greater than a first delay threshold and the packet loss rate is greater than a preset packet loss rate threshold reaches a first duration threshold, the ratio of the first delay threshold to the current delay is obtained. The ratio is used as an adjustment factor to reduce the size of the data packet or decrease the transmission rate.
4. The method according to claim 3, characterized in that, After reducing the data packet size or decreasing the transmission rate, the method further includes: When the duration of the delay being less than the second delay threshold reaches the second duration threshold, The reduced packet size or reduced transmission rate will be restored to its original value.
5. The method according to claim 2, wherein, The execution performance metrics include the load performance metrics of the first target node, wherein adjusting the data migration plan when the execution performance metrics meet the adaptive adjustment triggering conditions includes: When the load performance index of the first target node exceeds the load threshold, a second target node with a load performance index lower than the load threshold is located; and At least a portion of the source data allocated to the first target node in the allocation information is reassigned to the second target node.
6. The method according to claim 1, wherein, The data migration plan includes: Collect basic migration information for the source distributed database cluster and the target distributed database cluster; Based on the migration infrastructure information, the data migration plan is obtained.
7. The method according to claim 1, wherein, The step of migrating each source data shard to the allocated target node based on the allocation information includes: At the start of the migration, the initial data state of the source data fragments is determined; Based on the initial data state, the source data shards are migrated to the target node to obtain the target data shards; After the migration is completed, consistency verification is performed on the target data shards on the target node.
8. The method according to claim 7, wherein, The consistency check of the target data shards on the target node includes: Calculate the first hash value of the source data fragment in the initial data state; Calculate the second hash value of the target data fragment; Compare the first hash value and the second hash value; When the first hash value matches the second hash value, the target data fragment verification is deemed to have passed. When the first hash value is inconsistent with the second hash value, the target data fragment verification is determined to be unsuccessful, and the exception handling process is triggered.
9. The method according to claim 8, wherein, The exception handling process includes: Based on the results of failed verification, identify the type of anomaly; Retrieve the repair strategy corresponding to the anomaly type from the preset anomaly repair rule base; The repair strategy is executed to repair the target data fragments; The consistency check is then performed again on the repaired target data shards.
10. The method according to claim 1, wherein, The method further includes: During the execution of the data migration plan, data changes in the source distributed database cluster are monitored, and an incremental data change log is generated. After all source data shards in the source distributed database cluster have been migrated and consistency verification has passed, the data changes are applied to the corresponding nodes in the target distributed database cluster according to the incremental data change log.
11. The method according to claim 10, wherein, The method further includes: During the process of applying the data changes corresponding to the incremental data change log to the target distributed database cluster, the subsequent data changes generated by the source distributed database cluster are continuously acquired, and the amount of subsequent data changes to be applied is determined. When the amount of data exceeds a preset threshold, the subsequent data changes are applied to the corresponding nodes of the target distributed database cluster, and the subsequent data changes generated by the source distributed database cluster are continuously acquired during the application. After the application is completed, the amount of data is re-determined and the above application operation is repeated. When the amount of data decreases to or below the preset threshold, the subsequent data changes are applied to the corresponding nodes of the target distributed database cluster, and business traffic is connected to the target distributed database cluster during the application process.
12. A distributed database data migration device, characterized in that, The device includes: A data migration plan acquisition module is used to acquire a data migration plan, which includes: allocation information of each source data shard in the source distributed database cluster to a target node in the target distributed database cluster, and a parallel migration degree representing the number of source data shards undergoing concurrent migration; and A data migration plan execution module is used to execute the data migration plan, including: controlling the parallel migration of m source data shards, where m is a positive integer and m is not greater than the parallel migration degree; and migrating each source data shard to the allocated target node based on the allocation information, wherein, after the migration is completed, a target data shard corresponding to the source data shard is obtained at the target node.
13. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.
14. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.
15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.
Citation Information
Cited By
Data migration method, device and equipment and computer readable storage medium
CN121807816A