Data migration method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data migration method, device and equipment and a computer readable storage medium, which are applied to the technical field of computers, and comprise the following steps: sending a new copy notification instruction to an old main copy node of a partition to be migrated through a controller node, and determining a new copy node according to the new copy notification instruction; the snapshot data of the partition to be migrated are split into a plurality of data segments through the old main copy node, and the data segments are migrated to the new copy node in a progressive mode; when the data segments are migrated to the new copy node in a progressive mode, state updating is conducted on the data segments, and a differential synchronization strategy is executed on the write request according to the current state of the target data segment. According to the method, on the premise of ensuring the migration efficiency, efficient read-write delay is ensured in the migration process, and meanwhile, the use condition of system resources is greatly reduced, so that the complex requirements of a large-scale distributed environment are met.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data migration method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Existing distributed caching migration solutions generally employ a combination of full migration and incremental compensation. However, both approaches have significant drawbacks: full migration places immense pressure on the master node, directly causing a sharp increase in read / write latency during data migration, severely impacting the response efficiency of external services; moreover, the larger the data volume, the longer the migration takes, and the longer the negative impact on services lasts, significantly reducing the overall system responsiveness during the migration process. Incremental compensation relies on logs for data synchronization. During migration, partition data in a moving state can lead to difficulties in request forwarding or a large number of requests being cached, resulting in a surge in memory and I / O (input / output) resource consumption, thus compromising system stability.

[0003] Therefore, how to reduce the pressure and resource consumption on the master node during the distributed cache data migration process, while avoiding affecting business response efficiency and system stability, is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data migration method, apparatus, device and computer-readable storage medium, which solves the problems of high pressure and resource consumption of the master node during the migration of distributed cache data in the prior art, which affects the efficiency of business response and system stability.

[0005] To address the aforementioned technical problems, this invention provides a data migration method, comprising: The controller node sends a new replica notification instruction to the old primary replica node of the partition to be migrated, and the new replica node is determined according to the new replica notification instruction. The snapshot data of the partition to be migrated is split into multiple data segments by the old primary replica node, and the data segments are migrated to the new replica node in a progressive manner. When migrating the data segment to the new replica node in a progressive manner, the state of each data segment is updated, and a differentiated synchronization strategy is executed on the write request based on the current state of the target data segment. The new replica node is a target node used to receive and store new replica data of the partition to be migrated during the migration process. The new replica node includes a new primary replica node and a new secondary replica node. The target data segment is the data segment corresponding to the write request.

[0006] Optionally, when migrating the data segments to the new replica node in a progressive manner, the state of each data segment is updated, and a differentiated synchronization strategy is executed on write requests based on the current state of the target data segment, including: When the write request is received during the process of progressively migrating the data segment to the new replica node, the current state of the target data segment is determined. If the current state of the target data segment is unmigrated, the write request will be synchronized to the old slave replica node; the unmigrated state is the state in which the snapshot data of the target data segment has not yet started to migrate. If the current state of the target data segment is in the process of migration, the write request will be synchronized to the old slave replica node and the new replica node after the migration is completed; the state of migration is the state in which the snapshot data of the target data segment is being migrated. If the current state of the target data segment is a migrated state, then the write request is synchronized to the old slave replica node and the new replica node; the migrated state means that the snapshot data of the target data segment has been migrated to the new replica node.

[0007] Optionally, when migrating the data segments to the new replica node in a progressive manner, after updating the state of each data segment and executing a differentiated synchronization strategy for write requests based on the current state of the target data segment, the method further includes: The consistency of data between the new replica node and the old primary replica node is verified through the old primary replica node; If the verification fails, the data migration steps will be re-executed.

[0008] Optionally, if the verification is successful, it also includes: Lock the partition to be migrated, the old primary replica node, and the new primary replica node, and asynchronously synchronize the partition management table update log to all nodes in the cluster. After the new primary replica node applies the partition management table update log, it unlocks the partition to be migrated, and the new primary replica node takes over and processes the read and write requests of the partition to be migrated.

[0009] Optionally, it also includes: After the old primary replica node applies the partition management table update log, it forwards the residual requests to the new primary replica node and clears the local data of the partition to be migrated on the old primary replica node.

[0010] Optionally, the state includes an unmigrated state, a migrating state, and a migrated state.

[0011] Optionally, the snapshot data of the partition to be migrated is split into multiple data segments by the old primary replica node, and the data segments are migrated to the new replica node in a progressive manner, including: Step 1: Divide the snapshot data of the partition to be migrated into several independent data segments using the old master replica node; Step 2: Sequentially lock a preset number of first data segments that are not yet migrated through the old master replica node; Step 3: After the snapshot data migration of the first data segment is completed, unlock and mark it as migrated; Step 4: Execute Step 2 and Step 3 in sequence until all data segments of the partition to be migrated are synchronized.

[0012] The present invention also provides a data migration apparatus, comprising: The new replica node determination module is used to send a new replica notification instruction to the old primary replica node of the partition to be migrated through the controller node, and determine the new replica node according to the new replica notification instruction. The data splitting and migration module is used to split the snapshot data of the partition to be migrated into multiple data segments through the old primary replica node, and migrate the data segments to the new replica node in a progressive manner; The differentiated write request synchronization module is used to update the status of each data segment when the data segment is migrated to the new replica node in a progressive manner, and to execute a differentiated synchronization strategy for write requests based on the current status of the target data segment. The new replica node is a target node used to receive and store new replica data of the partition to be migrated during the migration process. The new replica node includes a new primary replica node and a new secondary replica node. The target data segment is the data segment corresponding to the write request.

[0013] The present invention also provides a data migration device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the data migration method as described above.

[0014] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the data migration method described above.

[0015] As can be seen, this invention sends a new replica notification instruction to the old primary replica node of the partition to be migrated through the controller node, and determines the new replica node based on the new replica notification instruction; the snapshot data of the partition to be migrated is split into multiple data segments through the old primary replica node, and the data segments are migrated to the new replica node in a progressive manner; when the data segments are migrated to the new replica node in a progressive manner, the status of each data segment is updated, and a differentiated synchronization strategy is executed on the write request based on the current status of the target data segment; wherein, the new replica node is used as the target node for receiving and storing the new replica data of the partition to be migrated during the migration process, and the new replica node includes a new primary replica node and a new secondary replica node; the target data segment is the data segment corresponding to the write request. This invention employs a segmented, progressive transmission mechanism to replace the traditional one-time full data transmission method. This effectively distributes the data transmission pressure on the old primary and replica nodes, reduces their instantaneous load peaks, and fundamentally avoids the problem of sharp increases in read and write latency during data migration, significantly improving the response efficiency of business requests during migration. Simultaneously, a differentiated write request synchronization strategy is designed for data segments in different migration states within a partition. By accurately matching the data segment state with the synchronization logic, it completely solves the problems of request forwarding difficulties and large-scale request cache accumulation caused by partition data being in a migration state in traditional incremental compensation schemes. This significantly reduces the additional consumption of memory and I / O resources, ensuring the system stability of the distributed cache cluster during the migration process.

[0016] In addition, the present invention also provides a data migration apparatus, device, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart of a data migration method provided in an embodiment of the present invention; Figure 2 An overall architecture diagram provided for an embodiment of the present invention; Figure 3 A flowchart illustrating a migration instruction scheduling method provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the master-slave replica synchronization process for write requests during migration, provided as an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a data migration device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a data migration device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Distributed caching systems are indispensable infrastructure components in modern large-scale internet applications and enterprise-level systems. These systems achieve distributed data storage and high-speed access by storing data in the memory of multiple nodes, and are widely used in high-concurrency scenarios such as social networks, e-commerce, financial transactions, and online games. As business scales up, the amount of data handled by distributed caching systems continuously increases. In such cases, the original data nodes can no longer handle the increased data volume, necessitating the expansion of nodes and the migration of data to balance the data across the entire cluster.

[0021] With the widespread adoption of distributed caching in large-scale clusters, data migration methods have evolved from simple to complex, and from full-downtime to highly available online methods. Currently, mainstream distributed caching systems generally employ a combination of full migration and incremental compensation for data migration. Full migration typically involves a one-time copy of a large amount of data from the source node to build an initial data view for the target node; while incremental compensation continuously records and replays new or updated operations during the migration process to ensure data consistency before and after the migration. This combined approach balances migration efficiency with uninterrupted business operations during migration, making it the mainstream model used by Redis Cluster, Codis (Codis distributed Redis proxy cluster), Dragonfly (Dragonfly distributed cache), and most self-developed caching systems.

[0022] With the widespread adoption of cloud computing and microservice architectures, the deployment scale and complexity of distributed caching systems are constantly increasing. The number of cluster nodes has expanded from a few to hundreds or even thousands, the number of data partitions has reached tens of thousands, and the memory capacity of a single node has grown from GB to hundreds of GB. In this large-scale deployment environment, traditional data migration solutions face severe challenges. While various data migration methods have been proposed and applied in existing distributed caching technologies, some significant drawbacks still limit their effectiveness in large-scale data environments. For example, full migration puts significant pressure on the master node, leading to a sharp increase in read / write latency during data migration, affecting the response efficiency of external services. Especially with large data volumes, the migration time is longer, and the impact on external services lasts longer, significantly reducing the system's responsiveness during data migration. Incremental compensation methods rely on log synchronization. During migration, if data in a particular partition is in motion, it can cause difficulties in request forwarding or lead to a large number of requests being cached, which also significantly increases memory and I / O consumption during migration, affecting system stability.

[0023] To address the aforementioned issues, this invention provides a distributed cache data migration method based on master-slave synchronization optimization. This method reduces read / write latency during migration while maintaining migration efficiency, significantly decreasing system resource usage and thus adapting to the complex requirements of large-scale distributed environments. This invention is a data migration scheme that coordinates master-slave synchronization and partition instruction scheduling. Its core idea is to reduce the impact on cluster read / write operations during migration by progressively migrating partitions, while simultaneously ensuring the proper backup of newly written data during migration through a master-slave synchronization mechanism.

[0024] Please refer to the details. Figure 1 , Figure 1 A flowchart illustrating a data migration method provided in an embodiment of the present invention. The method may include: S101: Send a new replica notification instruction to the old primary replica node of the partition to be migrated through the controller node, and determine the new replica node according to the new replica notification instruction.

[0025] The execution entity in this embodiment is a terminal. This embodiment does not limit the type of terminal, as long as it can complete the data migration operation. It should be noted that the primary replica node in this embodiment is used to handle read and write requests for the partition to be migrated and to initiate data migration. The new replica nodes are target nodes used to receive and store new replica data of the partition to be migrated during the migration process. The new replica nodes include a new primary replica node and new secondary replica nodes. In this step, the controller node notifies the old master replica node of the partition which nodes are the newly added replica nodes for that partition, i.e., the new replica nodes.

[0026] In this embodiment, nodes are divided into two categories: Controller Nodes, responsible for cluster metadata management and instruction scheduling during cluster migration; and Data Nodes, responsible for storing data partitions, executing read and write requests, and executing specific migration instructions. In a distributed caching cluster, partitions are the basic units of data storage and management. To ensure data reliability and availability, each partition is typically configured with a Master replica (handling read and write requests) and a Slave replica (used for backup and read request sharing). When partition data migration is triggered (e.g., node scaling up / down, load balancing, node failure recovery), the corresponding Master and Slave replicas will not remain fixed on the original node but may be allocated to other nodes in the cluster. This change in replica nodes is not random; the specific target node (i.e., the new replica node) is determined by the cluster's preset partitioning algorithm (such as consistent hashing, range partitioning, etc.). The partitioning algorithm, based on factors such as cluster node topology and load status, completes the location and allocation of replicas on new nodes to achieve overall cluster load balancing and high availability. For details, please refer to [link to relevant documentation]. Figure 2 The cluster architecture layer in the middle, Figure 2 This is an overall architecture diagram provided for an embodiment of the present invention. A master replica (master replica data) of a partition is deployed on a specific node, which may be a controller node or a data node; this node is the master replica node (master replica node) of that partition. The corresponding slave replicas (slave replica data) are deployed on other nodes, which may also be controller nodes or data nodes; these nodes are the slave replica nodes (slave replica nodes) of that partition. Generally, the master replica node and the slave replica node of the same partition are not the same node.

[0027] S102: Split the snapshot data of the partition to be migrated into multiple data segments through the old primary replica node, and migrate the data segments to the new replica node in a progressive manner.

[0028] This step involves data-level replica synchronization. The old master replica node (i.e., the old primary replica node) transmits / migrates the complete partition data to the designated new master replica node and / or new slave replica node in the distributed cache cluster. This completes the data initialization of the new replica node, ensuring it has the storage and service capabilities for the partition data. The migration of the partition's snapshot data is scheduled via migration commands, which are calculated by the controller node and responsible for scheduling the execution of each data node. It should be noted that the snapshot data in this embodiment is a complete and static data image generated by the old primary replica node for the partition or a single data segment to be migrated before / during the migration.

[0029] During the migration process, you can refer to Figure 2 In the storage layer, the old Master replica node splits the snapshot data of the partition into data segments and gradually migrates them to the new replica node. Specifically, during the data migration process of a partition, a partition is divided into multiple data segments, and only a preset number of segments are locked for transmission at a time. This gradual migration of data for a partition greatly reduces the impact on external read and write requests during the migration process compared to locking the entire partition for data transmission.

[0030] This embodiment does not limit the splitting method or the number of splits. For example, the old primary replica node first generates a full snapshot of the partition data, and then splits this snapshot data into multiple independent data segments according to preset sharding rules (such as data hash range, fixed data block size, key-value number threshold, etc.). Each data segment is marked with an initial Not Migrated state, completing the smallest granularity of data migration. This splitting achieves gradual data transmission, avoiding the instantaneous high load and sharp increase in read / write latency on the old primary replica node caused by a full one-time transmission. It also provides a foundation for subsequent differential write request synchronization based on data segment status, locking data segments in migration, precise verification, and retries, significantly improving the controllability, data consistency, and system stability of the migration process.

[0031] Furthermore, the above-mentioned method of splitting the snapshot data of the partition to be migrated into multiple data segments through the old primary replica node and migrating the data segments to the new replica node in a progressive manner can specifically include: Step 1: Dividing the snapshot data of the partition to be migrated into several independent data segments through the old primary replica node; Step 2: Locking a preset number (one or more) of the first data segments in the unmigrated state through the old primary replica node; Step 3: Unlocking and marking the first data segment as migrated after the snapshot data migration is completed; Step 4: Executing Step 2 and Step 3 in sequence until all data segments of the partition to be migrated are synchronized. This implementation uses data segmentation and batch locking migration instead of full one-time transmission. It can flexibly control the amount of data migrated in a single batch, avoiding read and write request blocking caused by instantaneous high load on the old master and replica nodes, and effectively reducing service latency during migration. The closed-loop operation of locking-migrating-unlocking-status marking for data segments in batches can accurately control the migration progress of each data segment. Combined with the subsequent differentiated write request synchronization strategy, it can avoid the consistency risk caused by data tampering during migration. It supports batch migration according to a preset number of data segments, and can dynamically adjust the batch size according to the current cluster load, improving the flexibility and adaptability of the migration strategy. At the same time, if the migration of a single data segment fails, only that batch needs to be retried, without rolling back all partition data, which greatly reduces the cost of migration retry and ensures the stability and availability of the distributed cache cluster during the migration process.

[0032] S103: When migrating data segments to new replica nodes in a progressive manner, update the status of each data segment and execute a differentiated synchronization strategy for write requests based on the current status of the target data segment.

[0033] During incremental migration, the state of each data segment within a partition may change as the migration progresses, necessitating state updates for each segment. Each data segment has three states: Migrated (migrated): Indicates that the snapshot data for this data segment has been synchronized to the newly added replica nodes. Migrating: Indicates that the snapshot data for this data segment is being synchronized. Not Migrated: Indicates that the snapshot data for this data segment has not yet begun synchronization. When a data segment begins transmission, it is marked from the Not Migrated state to the Migrating state; when the data synchronization for this data segment is complete, it is marked as Migrated.

[0034] Since updates to the partition management table are performed on a partition-by-partition basis, the distributed cache cluster still relies on the old Master replica nodes to handle read and write requests for that partition until the data in the entire partition is synchronized. When a partition receives a write request during the migration process, a differentiated synchronization strategy needs to be applied to the write request based on the current state of the data segment corresponding to the write request (i.e., the target data segment).

[0035] Furthermore, when migrating data segments to new replica nodes in a progressive manner, the status of each data segment is updated, and a differentiated synchronization strategy is executed on write requests based on the current status of the target data segment. Specifically, this may include: when a write request is received during the progressive migration of a data segment to a new replica node, determining the current status of the target data segment; if the current status of the target data segment is "not migrated," then the write request is synchronized to the old slave replica node; the "not migrated" state is the state where the snapshot data of the target data segment has not yet started migrating; if the current status of the target data segment is "migrating," then the write request is synchronized to both the old slave replica node and the new replica node after the migration is completed; the "migrating" state is the state where the snapshot data of the target data segment is in the process of migrating; if the current status of the target data segment is "migrated," then the write request is synchronized to both the old slave replica node and the new replica node; the "migrated" state is the state where the snapshot data of the target data segment has been migrated to the new replica node. Specifically, a partition may have some data segments that have been synchronized and some that have not. To ensure data consistency, the master-slave replica synchronization of read and write requests during partition migration is handled as follows: (1) If the write request falls in a data segment in the Not Migrated state, the write request only needs to be synchronized to the old Slave replica node. This is to prevent the old Slave replica node from taking over the Master role while holding data consistent with the Master if the old Master replica node crashes during the migration process, thus preventing data loss. For the new replica node, this write request is already included in the data snapshot of the subsequent data segment synchronization. (2) If the write request falls in a data segment in the Migrating state, the data segment is in a locked state. After the transmission is completed, it will be marked as Migrated and the Migrated request processing logic will be executed. (3) If the write request falls in a data segment in the Migrated state, the master-slave replica synchronization of the request needs to be performed on the new replica node of the partition. Otherwise, inconsistency of replica data will occur. Because the data of this data segment will no longer be synchronized with the data snapshot, it can also be further synchronized to the old slave replica node.

[0036] Furthermore, the above-mentioned process of progressively migrating data segments to new replica nodes, including updating the state of each data segment and implementing differentiated synchronization strategies for write requests based on the current state of the target data segment, can further include: performing consistency checks on the data between the new and old replica nodes using the old master replica node; if the check fails, the data migration steps are re-executed. This embodiment also performs data integrity checks. After the snapshot data of the partition is synchronized, the consistency between the data on the new replica node and the old master replica node is checked. If they are inconsistent, it indicates that data loss or master-slave replica synchronization failure occurred during the transmission / migration process, and the data migration for that partition needs to be repeated.

[0037] Furthermore, if the verification is successful, the process may also include: locking the partition to be migrated, the old master replica node, and the new master replica node; asynchronously synchronizing the partition management table update log to all nodes in the cluster; and unlocking the partition to be migrated after the new master replica node applies the partition management table update log, allowing the new master replica node to take over and process read and write requests for the partition to be migrated. Upon successful verification, the cluster metadata is updated: the controller node modifies the unified partition management table of the distributed storage cluster, officially updating the node information of the Master and Slave replicas corresponding to that partition to the new node information after migration. Specifically, this can involve temporarily locking the partition, using the Raft algorithm (a distributed consensus algorithm) to asynchronously synchronize the partition management table update log to all nodes in the cluster. Once the new Master replica node applies the partition management table update log, it unlocks the partition and begins processing read and write requests normally. Considering that asynchronous synchronization may lead to temporary inconsistencies in the partition management tables of different nodes, the old and new Master replica nodes of the partition need to be temporarily locked before log synchronization (i.e., the cluster metadata update operation). Meanwhile, to ensure data consistency and avoid read / write conflicts, all read / write requests for that partition within the cluster must be handled by the old Master replica node before the controller node completes the partition management table update, and cannot be directly routed to the new replica node.

[0038] Furthermore, the above may also include: after the old master replica node applies the partition management table update log, it forwards the remaining requests to the new master replica node and clears the local data of the partition to be migrated on the old master replica node. In this embodiment, after the old master replica node applies the log and unlocks, the requests remaining on the old master replica node will be forwarded to the new master replica node. At the same time, after all the old replica nodes of the partition apply the log, if they are no longer replica nodes of the partition, they also need to clear the data of the partition and release the memory.

[0039] The data migration method provided in this embodiment of the invention proceeds as follows: S101: The controller node sends a new replica notification instruction to the old primary replica node of the partition to be migrated, and determines the new replica node according to the new replica notification instruction; S102: The old primary replica node splits the snapshot data of the partition to be migrated into multiple data segments, and migrates the data segments to the new replica node in a progressive manner; S103: When migrating the data segments to the new replica node in a progressive manner, the status of each data segment is updated, and a differentiated synchronization strategy is executed on the write request according to the current status of the target data segment. This method employs a segmented, progressive transmission mechanism to replace the traditional one-time full data transmission method. This effectively distributes the data transmission pressure on the old primary and replica nodes, reducing their instantaneous load peaks and fundamentally avoiding the problem of sharp increases in read / write latency during data migration. This significantly improves the response efficiency of business requests during migration. Simultaneously, a differentiated write request synchronization strategy is designed for data segments in different migration states within a partition. By accurately matching the data segment state with the synchronization logic, it completely solves the problems of request forwarding difficulties and large-scale request cache accumulation caused by partition data being in a migration state, which are common in traditional incremental compensation schemes. This significantly reduces the additional consumption of memory and I / O resources, ensuring the system stability of the distributed cache cluster during migration. Furthermore, while ensuring migration efficiency, it achieves high read / write latency during migration, while also greatly reducing the use of system resources, thus adapting to the complex needs of large-scale distributed environments. Moreover, the Raft algorithm's asynchronous update of the partition management table and the short-term locking mechanism avoid the resource consumption of metadata synchronization and prevent request forwarding chaos during migration, reducing the additional occupation of memory and I / O resources.

[0040] For a clearer understanding of this invention, please refer to the following details. Figure 3 , Figure 3 A flowchart illustrating a migration instruction scheduling method provided in this embodiment of the invention may specifically include: The controller node initiates a command: The controller node sends a command to Data Node 1 to specify the new replica node (i.e., New Node A / B) corresponding to the old Master / Slave replica of a certain partition on the node. The old master and slave nodes prepare data: Data Node 1 (currently the old master node for this partition) prepares all the data for the partition to be migrated; Data transfer between the old primary and secondary nodes: Data Node 1 will synchronously transfer all data to the new replica nodes (New NodeA / B). The controller node sends a verification command: The controller node sends a command to data node 1, requesting it to perform a consistency verification between the full data of the new replica node and its own data; The old master and slave nodes perform a consistency check: Data Node 1 performs a consistency check on the replica data on New Node A / B to ensure that the data on the new node is completely consistent with the old node; The controller node notifies Redis (an in-memory data structure store used as a database, cache, and message broker) to update: The controller node sends a command to Redis to update the replica node information corresponding to the partition (replacing the old node with the new node A / B). Data cleanup on original / new nodes: Data Node 1 deletes the local data of this partition, while New Node A / B completes the final confirmation of the replica data, thus completing the migration.

[0041] Figure 4 A flowchart illustrating the master-slave replica synchronization process for write requests during migration, provided as an embodiment of the present invention, may specifically include: Data Node 1 is the old primary replica node for partition 1, responsible for receiving external requests and managing data segment migration. DataNodes 2, 3, and 4 are the new replica nodes for partition 1 (with Data Node 3 eventually becoming the new primary replica node). DataNode 1 splits partition 1 into three data segments with the following states: Segment 0 (Not Migrated): Not migrated, only requests synchronization with the old secondary replica node (Data Node 2); Segment 1 (Migrating): Migrating in progress, in a locked state; Segment 2 (Migrated): Migrated, requires synchronization with the new replica nodes (Data Nodes 3 and 4). External requests are sent to Data Node 1, which processes them according to the segment status: Segment 0 requests: only synchronized to Data Node 2; Segment 2 requests: synchronized to Data Nodes 3 and 4. After a data segment migration is complete (e.g., segment 2), it will synchronize with the new nodes according to primary-secondary synchronization to ensure data consistency. Once all data segments have been migrated, the primary replica node of partition 1 will switch to Data Node 3, completing the migration of the entire partition.

[0042] The data migration apparatus provided in the embodiments of the present invention will be described below. The data migration apparatus described below and the data migration method described above can be referred to each other.

[0043] Please refer to the details. Figure 5 , Figure 5 A schematic diagram of a data migration device provided in an embodiment of the present invention may include: The new replica node determination module 100 is used to send a new replica notification instruction to the old primary replica node of the partition to be migrated through the controller node, and determine the new replica node according to the new replica notification instruction. The data splitting and migration module 200 is used to split the snapshot data of the partition to be migrated into multiple data segments through the old primary replica node, and migrate the data segments to the new replica node in a progressive manner; The differentiated write request synchronization module 300 is used to update the status of each data segment when the data segment is migrated to the new replica node in a progressive manner, and to execute a differentiated synchronization strategy for write requests based on the current status of the target data segment. The new replica node is a target node used to receive and store new replica data of the partition to be migrated during the migration process. The new replica node includes a new primary replica node and a new secondary replica node. The target data segment is the data segment corresponding to the write request.

[0044] Furthermore, based on any of the above embodiments, the differentiated write request synchronization module 300 may include: A status determination unit is used to determine the current status of the target data segment when a write request is received during the process of migrating the data segment to the new replica node in a progressive manner. The first processing unit is configured to synchronize the write request to the old slave replica node if the current state of the target data segment is an unmigrated state; the unmigrated state is the state in which the snapshot data of the target data segment has not yet started to migrate. The second processing unit is configured to synchronize the write request to the old slave replica node and the new replica node after the migration is completed if the current state of the target data segment is in the migration process; the migration process refers to the state in which the snapshot data of the target data segment is being migrated. The third processing unit is configured to synchronize the write request to the old slave replica node and the new replica node if the current state of the target data segment is a migrated state; the migrated state is the state in which the snapshot data of the target data segment has been migrated to the new replica node.

[0045] Furthermore, based on any of the above embodiments, the data migration apparatus may further include: The consistency verification module is used to update the status of each data segment when the data segment is migrated to the new replica node in a progressive manner, and after executing a differentiated synchronization strategy on the write request according to the current status of the target data segment, perform consistency verification on the data of the new replica node and the old primary replica node through the old primary replica node. The first result module is used to re-execute the data migration steps if the verification fails.

[0046] Furthermore, based on any of the above embodiments, the data migration apparatus may further include: The second processing module is used to lock the partition to be migrated, the old primary replica node, and the new primary replica node if the verification is successful, and asynchronously synchronize the partition management table update log to all nodes in the cluster; when the new primary replica node applies the partition management table update log, it unlocks the partition to be migrated, and the new primary replica node takes over and processes the read and write requests of the partition to be migrated.

[0047] Furthermore, based on any of the above embodiments, the data migration apparatus may further include: The third processing module is used to forward residual requests to the new primary replica node and clear the local data of the partition to be migrated on the old primary replica node after the old primary replica node applies the partition management table update log.

[0048] Furthermore, based on any of the above embodiments, the state includes an unmigrated state, a migrating state, and a migrated state.

[0049] Furthermore, based on any of the above embodiments, the data splitting and migration module 200 may include: The first execution unit is used to execute step 1: dividing the snapshot data of the partition to be migrated into several independent data segments through the old master replica node; The second execution unit is used to execute step 2: sequentially locking a preset number of first data segments that are not migrated through the old master replica node; The third execution unit is used to execute step 3: after the snapshot data migration of the first data segment is completed, unlock and mark it as migrated; The fourth execution unit is used to execute step 4: execute step 2 and step 3 in sequence until all data segments of the partition to be migrated are synchronized.

[0050] It should be noted that the order of the modules and units in the above data migration device can be changed without affecting the logic.

[0051] The data migration apparatus provided in this embodiment of the invention includes a new replica node determination module 100, which sends a new replica notification instruction to the old primary replica node of the partition to be migrated via a controller node, and determines the new replica node based on the new replica notification instruction; a data splitting and migration module 200, which splits the snapshot data of the partition to be migrated into multiple data segments via the old primary replica node, and migrates the data segments to the new replica node in a progressive manner; and a differentiated write request synchronization module 300, which updates the status of each data segment when the data segments are migrated to the new replica node in a progressive manner, and executes a differentiated synchronization strategy for write requests based on the current status of the target data segment; wherein, the new replica node is a target node used to receive and store the new replica data of the partition to be migrated during the migration process, and the new replica node includes a new primary replica node and a new secondary replica node; the target data segment is the data segment corresponding to the write request. This device employs a segmented, progressive transmission mechanism to replace the traditional one-time full data transmission method. This effectively distributes the data transmission pressure on the old primary and replica nodes, reduces their instantaneous load peaks, and fundamentally avoids the problem of sharp increases in read and write latency during data migration, significantly improving the response efficiency of business requests during migration. Simultaneously, a differentiated write request synchronization strategy is designed for data segments in different migration states within a partition. By accurately matching the data segment state with the synchronization logic, it completely solves the problems of request forwarding difficulties and large-scale request cache accumulation caused by partition data being in a migration state, which are common in traditional incremental compensation schemes. This significantly reduces the additional consumption of memory and I / O resources, ensuring the system stability of the distributed cache cluster during the migration process.

[0052] The data migration device provided in the embodiments of the present invention will be described below. The data migration device described below and the data migration method described above can be referred to each other.

[0053] Please refer to Figure 6 , Figure 6 A schematic diagram of a data migration device provided in an embodiment of the present invention may include: Memory 10 is used to store computer programs; Processor 20 is used to execute computer programs to implement the data migration method described above.

[0054] The memory 10, processor 20, and communication interface 31 all communicate with each other through the communication bus 32.

[0055] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions: The controller node sends a new replica notification instruction to the old primary replica node of the partition to be migrated, and determines the new replica node based on the new replica notification instruction. The snapshot data of the partition to be migrated is split into multiple data segments by using the old primary replica node, and the data segments are migrated to the new replica node in a progressive manner; When migrating data segments to new replica nodes in a progressive manner, the state of each data segment is updated, and a differentiated synchronization strategy is executed on write requests based on the current state of the target data segment. Among them, the new replica node is the target node used to receive and store the new replica data of the partition to be migrated during the migration process. The new replica node includes the new primary replica node and the new secondary replica node; the target data segment is the data segment corresponding to the write request.

[0056] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0057] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0058] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0059] Communication interface 31 can be an interface for the communication module, used to connect with other devices or systems.

[0060] Of course, it should be noted that, Figure 6 The structure shown does not constitute a limitation on the data migration device in the embodiments of the present invention. In practical applications, the data migration device may include more than Figure 6 More or fewer components as shown, or combinations of certain components.

[0061] The following describes the computer-readable storage medium provided in the embodiments of the present invention. The computer-readable storage medium described below can be referred to in correspondence with the data migration method described above.

[0062] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data migration method.

[0063] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0065] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0066] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0067] The above provides a detailed description of a data migration method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data migration method, characterized in that, include: The controller node sends a new replica notification instruction to the old primary replica node of the partition to be migrated, and the new replica node is determined according to the new replica notification instruction. The snapshot data of the partition to be migrated is split into multiple data segments by the old primary replica node, and the data segments are migrated to the new replica node in a progressive manner. When migrating the data segment to the new replica node in a progressive manner, the state of each data segment is updated, and a differentiated synchronization strategy is executed on the write request based on the current state of the target data segment. The new replica node is a target node used to receive and store new replica data of the partition to be migrated during the migration process. The new replica node includes a new primary replica node and a new secondary replica node. The target data segment is the data segment corresponding to the write request.

2. The data migration method according to claim 1, characterized in that, When migrating the data segments to the new replica node in a progressive manner, the state of each data segment is updated, and a differentiated synchronization strategy is executed on write requests based on the current state of the target data segment, including: When the write request is received during the process of progressively migrating the data segment to the new replica node, the current state of the target data segment is determined. If the current state of the target data segment is unmigrated, the write request will be synchronized to the old slave replica node; the unmigrated state is the state in which the snapshot data of the target data segment has not yet started to migrate. If the current state of the target data segment is in the process of migration, the write request will be synchronized to the old slave replica node and the new replica node after the migration is completed; the state of migration is the state in which the snapshot data of the target data segment is being migrated. If the current state of the target data segment is a migrated state, then the write request is synchronized to the old slave replica node and the new replica node; the migrated state means that the snapshot data of the target data segment has been migrated to the new replica node.

3. The data migration method according to claim 1, characterized in that, When migrating the data segments to the new replica node in a progressive manner, after updating the state of each data segment and executing a differentiated synchronization strategy on write requests based on the current state of the target data segment, the process further includes: The consistency of data between the new replica node and the old primary replica node is verified through the old primary replica node; If the verification fails, the data migration steps will be re-executed.

4. The data migration method according to claim 3, characterized in that, If the verification is successful, it also includes: Lock the partition to be migrated, the old primary replica node, and the new primary replica node, and asynchronously synchronize the partition management table update log to all nodes in the cluster. After the new primary replica node applies the partition management table update log, it unlocks the partition to be migrated, and the new primary replica node takes over and processes the read and write requests of the partition to be migrated.

5. The data migration method according to claim 4, characterized in that, Also includes: After the old primary replica node applies the partition management table update log, it forwards the residual requests to the new primary replica node and clears the local data of the partition to be migrated on the old primary replica node.

6. The data migration method according to claim 1, characterized in that, The states include non-migrated, migrating, and migrated.

7. The data migration method according to any one of claims 1 to 6, characterized in that, The snapshot data of the partition to be migrated is split into multiple data segments using the old primary replica node, and the data segments are migrated to the new replica node in a progressive manner, including: Step 1: Divide the snapshot data of the partition to be migrated into several independent data segments using the old master replica node; Step 2: Sequentially lock a preset number of first data segments that are not yet migrated through the old master replica node; Step 3: After the snapshot data migration of the first data segment is completed, unlock and mark it as migrated; Step 4: Execute Step 2 and Step 3 in sequence until all data segments of the partition to be migrated are synchronized.

8. A data migration device, characterized in that, include: The new replica node determination module is used to send a new replica notification instruction to the old primary replica node of the partition to be migrated through the controller node, and determine the new replica node according to the new replica notification instruction. The data splitting and migration module is used to split the snapshot data of the partition to be migrated into multiple data segments through the old primary replica node, and migrate the data segments to the new replica node in a progressive manner; The differentiated write request synchronization module is used to update the status of each data segment when the data segment is migrated to the new replica node in a progressive manner, and to execute a differentiated synchronization strategy for write requests based on the current status of the target data segment. The new replica node is a target node used to receive and store new replica data of the partition to be migrated during the migration process. The new replica node includes a new primary replica node and a new secondary replica node. The target data segment is the data segment corresponding to the write request.

9. A data migration device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data migration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the data migration method as described in any one of claims 1 to 7.

Images