Data migration method, device and program product

Abstract

The application discloses a data migration method, device and program product. It relates to the field of distributed or financial technology, and comprises the following steps: marking a data slot of to-be-migrated data in a source cluster to obtain a marked data slot; intercepting a write operation on the marked data slot and converting the write operation into a structured log to obtain a target log; adding the target log to a target queue and performing a playback operation on the write operation in a target cluster based on the target queue to obtain an updated target cluster, wherein the playback operation is used for writing data associated with the write operation into the target cluster; detecting whether the source cluster and the updated target cluster meet a traffic switching condition to obtain a detection result; and migrating access traffic from the source cluster to the updated target cluster when the detection result indicates that the source cluster and the updated target cluster meet the traffic switching condition. Through the application, the problem of high downtime risk in cluster migration by using a migration tool in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of distributed or financial technology, and more specifically, to a data migration method, apparatus, and program product. Background Technology

[0002] In related technologies, the migration key command of the native remote dictionary service migration tool is used to migrate data to the remote dictionary service cluster. When migrating large keys, write operations are blocked, which can lead to service interruption. In addition, key-level serial migration is slow, and TB-level data migration takes more than 24 hours, and a full re-migration is required after a failure.

[0003] There is currently no effective solution to the problem of high downtime risk when using migration tools for cluster migration in related technologies. Summary of the Invention

[0004] The main purpose of this application is to provide a data migration method, apparatus, and program product to solve the problem of high downtime risk when using migration tools for cluster migration in related technologies.

[0005] To achieve the above objectives, according to one aspect of this application, a data migration method is provided. The method includes: responding to a data migration request, marking data slots in a source cluster containing data to be migrated, obtaining marked data slots, wherein the data migration request is used to request the migration of the data to be migrated from the source cluster to a target cluster; intercepting write operations on the marked data slots and converting the write operations into structured logs, obtaining a target log; adding the target log to a target queue, and performing a replay operation on the write operations in the target cluster based on the target queue, obtaining an updated target cluster, wherein the replay operation is used to write data associated with the write operations into the target cluster; detecting whether the source cluster and the updated target cluster meet traffic switching conditions, obtaining a detection result; and if the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions, migrating access traffic from the source cluster to the updated target cluster.

[0006] Further, detecting whether the source cluster and the updated target cluster meet the traffic switching conditions and obtaining the detection result includes: acquiring a data snapshot of the source cluster to obtain a first snapshot, wherein the data snapshot includes the result obtained after processing the data slot snapshot based on the Cyclic Redundancy Check algorithm; acquiring a data snapshot of the updated target cluster to obtain a second snapshot; performing a consistency check on the first snapshot and the second snapshot to obtain a check result; acquiring the playback delay of the playback operation; and detecting whether the check result and the playback delay meet the traffic switching conditions to obtain the detection result.

[0007] Further, the traffic switching conditions include: the data of the source cluster and the updated target cluster are consistent, and the replay delay of the replay operation is less than a preset threshold. The detection result includes: if the verification result indicates that the first snapshot and the second snapshot are consistent, and the replay delay is less than the preset threshold, then the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions; if the verification result indicates that the first snapshot and the second snapshot are inconsistent, and / or the replay delay is not less than the preset threshold, then the detection result indicates that the source cluster and the updated target cluster do not meet the traffic switching conditions.

[0008] Furthermore, the data migration method also includes: if the verification result indicates that the first snapshot and the second snapshot are inconsistent, replaying the write operation in the target cluster based on the target queue.

[0009] Further, adding the target log to the target queue includes: determining whether the data volume of the target log is greater than a preset data volume threshold, and obtaining a determination result; if the determination result indicates that the data volume of the target log is greater than the preset data volume threshold, splitting the target log into N log fragments, where N is a positive integer; and adding the N log fragments to the target queue.

[0010] Furthermore, based on the target queue, the write operation is replayed in the target cluster to obtain an updated target cluster, including: starting M threads to concurrently pull logs from the target queue, where M is a positive integer; and in the target cluster, the logs pulled by the M threads are replayed based on a conflict-free replication data type algorithm to obtain the updated target cluster.

[0011] Further, before intercepting write operations to the marked data slots and converting the write operations into structured logs to obtain the target logs, the process includes: binding the target address to the source cluster, wherein the target address includes: the network address accessed by the access traffic; migrating the access traffic from the source cluster to the updated target cluster, including: binding the target address to the updated target cluster.

[0012] Furthermore, the fields of the target log include: opcode, key value, and timestamp.

[0013] To achieve the above objectives, according to another aspect of this application, a data migration apparatus is provided. The apparatus includes: a marking unit, configured to, in response to a data migration request, mark data slots in a source cluster containing data to be migrated, obtaining marked data slots, wherein the data migration request is used to request the migration of the data to be migrated from the source cluster to a target cluster; a first processing unit, configured to intercept write operations on the marked data slots and convert the write operations into structured logs, obtaining a target log; a second processing unit, configured to add the target log to a target queue and, based on the target queue, perform a replay operation on the write operations in the target cluster, obtaining an updated target cluster, wherein the replay operation is used to write data associated with the write operations into the target cluster; a detection unit, configured to detect whether the source cluster and the updated target cluster meet traffic switching conditions, obtaining a detection result; and a migration unit, configured to, if the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions, migrate access traffic from the source cluster to the updated target cluster.

[0014] Further, the detection unit includes: a first acquisition subunit, used to acquire a data snapshot of the source cluster to obtain a first snapshot, wherein the data snapshot includes the result obtained after processing the snapshot of the data slot based on the Cyclic Redundancy Check algorithm; a second acquisition subunit, used to acquire a data snapshot of the updated target cluster to obtain a second snapshot; a verification subunit, used to perform consistency verification on the first snapshot and the second snapshot to obtain a verification result; a third acquisition subunit, used to acquire the replay delay of the replay operation; and a detection subunit, used to detect whether the verification result and the replay delay meet the traffic switching conditions to obtain the detection result.

[0015] Further, the traffic switching conditions include: the data of the source cluster and the updated target cluster are consistent, and the playback delay of the playback operation is less than a preset threshold. The detection subunit includes: a first processing module, used to determine that the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions when the verification result indicates that the first snapshot and the second snapshot are consistent and the playback delay is less than the preset threshold; and a determining module, used to determine that the detection result indicates that the source cluster and the updated target cluster do not meet the traffic switching conditions when the verification result indicates that the first snapshot and the second snapshot are inconsistent, and / or when the playback delay is not less than the preset threshold.

[0016] Furthermore, the data migration apparatus also includes a replay unit, configured to replay the write operation in the target cluster based on the target queue if the verification result indicates that the first snapshot and the second snapshot are inconsistent.

[0017] Further, the second processing unit includes: a judgment subunit, used to judge whether the data volume of the target log is greater than a preset data volume threshold, and obtain a judgment result; a sharding subunit, used to shard the target log to obtain N log shards when the judgment result indicates that the data volume of the target log is greater than the preset data volume threshold, where N is a positive integer; and an adding subunit, used to add the N log shards to the target queue.

[0018] Furthermore, the second processing unit includes: a startup subunit, used to start M threads to concurrently pull logs from the target queue, where M is a positive integer; and a replay subunit, used to perform a replay operation on the logs pulled by the M threads in the target cluster based on a conflict-free replication data type algorithm, to obtain the updated target cluster.

[0019] Furthermore, the data migration device further includes: a binding subunit, used to bind the target address to the source cluster before intercepting write operations on the marked data slots and converting the write operations into structured logs to obtain the target logs, wherein the target address includes: the network address accessed by the access traffic; the migration unit includes: a binding subunit, used to bind the target address to the updated target cluster.

[0020] Furthermore, the fields of the target log include: opcode, key value, and timestamp.

[0021] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the data migration method.

[0022] According to another aspect of this application, an electronic device is provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the data migration method during runtime.

[0023] According to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data migration method.

[0024] In this application, the following method is adopted: In response to a data migration request, the data slots of the data to be migrated in the source cluster are marked to obtain the marked data slots. The data migration request is used to request the migration of the data to be migrated from the source cluster to the target cluster. Write operations on the marked data slots are intercepted, and the write operations are converted into structured logs to obtain the target logs. The target logs are added to a target queue, and based on the target queue, the write operations are replayed in the target cluster to obtain the updated target cluster. The replay operation is used to write the data associated with the write operations to the target cluster. The source cluster and the updated target cluster are checked to see if they meet the traffic switching conditions, and the detection result is obtained. If the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions, the access traffic is migrated from the source cluster to the updated target cluster. This solves the technical problem of high downtime risk associated with cluster migration using migration tools in related technologies. In this application, data migration is performed by bidirectional writing to the source and target clusters through message queues and incremental logs, achieving zero-downtime migration and thus improving the efficiency of cluster data migration. Attached Figure Description

[0025] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data migration method is shown.

[0027] Figure 2 This is a flowchart of a data migration method provided according to an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of a data migration system provided according to an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the data migration process provided according to an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of another data migration process provided according to an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of a data migration apparatus provided according to an embodiment of this application;

[0032] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] It should be noted that the data migration method and apparatus in this application can be used in the field of fintech for the migration of cluster data, and can also be used in any field other than fintech for the migration of cluster data. This application does not limit the application field of the data migration method and apparatus.

[0036] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0037] Cyclic Redundancy Check (CRC / CRC64): In the scenario of remote dictionary service cluster migration, CRC and CRC64 refer to the core algorithms used for data consistency verification. By generating a fixed-length checksum, the data consistency between the source cluster and the target cluster during the migration process is verified. In the embodiment of this application, the checksum of the data slot (i.e., Slot) snapshot can be generated by cyclic redundancy check.

[0038] Dual write: The proxy layer forwards write requests to both the source and target clusters simultaneously, ensuring strong data consistency during migration.

[0039] Incremental logs: record structured data streams (including opcodes, key hashes, serialized values, and timestamps) of write operations to the remote dictionary service for asynchronous synchronization.

[0040] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) collected in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with relevant laws and standards, take necessary security measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0041] Example 1

[0042] According to an embodiment of this application, a method embodiment for data migration is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a data migration method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data migration method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned data migration method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0047] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] Under the aforementioned operating environment, this application provides the following: Figure 2 The data migration method shown. Figure 2 This is a flowchart of the data migration method according to Embodiment 1 of this application.

[0049] Step S201: In response to the data migration request, mark the data slots of the data to be migrated in the source cluster to obtain the marked data slots. The data migration request is used to request the migration of the data to be migrated in the source cluster to the target cluster.

[0050] The data migration request mentioned above can be used to request the movement of specific data from the source cluster (i.e., the currently running cluster) to the target cluster (the cluster that will become the new operating environment).

[0051] The types of the source cluster and target cluster mentioned above can include: distributed clusters, for example, the source cluster and target cluster can be remote dictionary service clusters.

[0052] In this embodiment, all data slots in the source cluster can be marked as "pending migration" and the target address (e.g., virtual IP) can be bound to the source cluster. Application traffic accesses the source cluster through the virtual IP. In an optional example, the virtual IP can be "VIP: 10.0.0.1".

[0053] In one alternative example, before data migration, all data in the source cluster can be copied to the target cluster. Then, incremental logs of write operations to the source cluster can be collected, and the write operations involved in the incremental logs can be replayed in the target cluster.

[0054] Step S202: Intercept write operations to the marked data slots and convert the write operations into structured logs to obtain the target log.

[0055] In this embodiment, write operations (e.g., SET / DEL / INCR) can be intercepted using source cluster kernel hooks to generate structured logs and obtain the target log. Here, the SET operation represents a command used to set key-value pairs, the DEL operation represents a command used to delete one or more specified keys, and the INCR operation represents an atomic increment operation used to increment the value stored in the key by 1.

[0056] If the value of a single target log (which can be defined as Value here) exceeds a predetermined size (for example, the log size exceeds 1MB), the target log can be sharded to obtain multiple log shards, and the shard identifier (e.g., shard ID) of each log shard can be appended to the key hash.

[0057] Step S203: Add the target log to the target queue, and based on the target queue, perform a replay operation on the write operation in the target cluster to obtain the updated target cluster. The replay operation is used to write the data associated with the write operation to the target cluster.

[0058] The target queue mentioned above may include the message queue of the stream processing platform. In this embodiment, target logs can be written to the target queue, and multiple consumer threads can be started to concurrently pull logs from the target queue for replay operations on the target cluster. It should be noted that during the replay operation, out-of-order conflicts can also be resolved using a conflict-free replicated data type (CRDT) algorithm.

[0059] Step S204: Detect whether the source cluster and the updated target cluster meet the traffic switching conditions and obtain the detection results.

[0060] The traffic switching conditions may include, but are not limited to: the data of the source cluster and the updated target cluster are consistent, and the playback delay of the playback operation is less than a preset threshold. In this embodiment, it is possible to detect whether the source cluster and the updated target cluster meet the traffic switching conditions and obtain the detection result.

[0061] Step S205: If the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions, the access traffic is migrated from the source cluster to the updated target cluster.

[0062] If the test results indicate that the source cluster and the updated target cluster meet the traffic switching conditions, the write to the source cluster can be turned off, and all access traffic can be switched to the target cluster. After that, the migrated data slot data can be deleted from the source cluster.

[0063] In one alternative example, this embodiment can perform migration rollback. For instance, by using a reverse synchronization window to synchronize logs from the target cluster to the source cluster, a rollback at the second level can be achieved in case of an anomaly during the migration process.

[0064] Through the above steps, in this embodiment, data migration is performed by bidirectional writing to the source and target clusters using message queues and incremental logs, achieving zero-downtime migration and thus improving the efficiency of cluster data migration. This solves the technical problem of high downtime risk when using migration tools for cluster migration in related technologies.

[0065] Optionally, in the data migration method provided in this application embodiment, detecting whether the source cluster and the updated target cluster meet the traffic switching conditions and obtaining the detection result includes: obtaining a data snapshot of the source cluster to obtain a first snapshot, wherein the data snapshot includes: the result obtained after processing the data slot snapshot based on the cyclic redundancy check algorithm; obtaining a data snapshot of the updated target cluster to obtain a second snapshot; performing a consistency check on the first snapshot and the second snapshot to obtain a check result; obtaining the playback delay of the playback operation; and detecting whether the check result and the playback delay meet the traffic switching conditions to obtain the detection result.

[0066] The first snapshot mentioned above can be a snapshot of the source cluster. This first snapshot can include a static view of all data in the source cluster, and can be either a full snapshot or an incremental snapshot. In this embodiment, a first snapshot of the source cluster can be obtained by calculating the data slot snapshot of the source cluster based on a cyclic redundancy check (CRC) algorithm. This first snapshot allows for rapid verification of data block integrity. The second snapshot can include: calculating the snapshot of the updated target cluster based on a CRC algorithm. After the target cluster undergoes a series of data synchronization and update operations, a second snapshot can be captured. By using both the first and second snapshots, it is possible to check whether the updated target cluster maintains data consistency with the source cluster.

[0067] For example, the first snapshot of the source cluster and the second snapshot of the updated target cluster can be compared to confirm whether they are completely consistent at the data level. If the first snapshot and the second snapshot are consistent, it can be determined that the calculation results of the cyclic redundancy check algorithm for the data slot snapshots of the source cluster and the updated target cluster are the same, and then it can be determined that the data of the source cluster and the data of the updated target cluster are consistent; otherwise, it can be determined that there is a problem in the process of synchronizing the incremental log data to the target cluster.

[0068] The replay latency of a replay operation can refer to the time required for incremental logs to be received and replayed into actual database operations in the target cluster. Replay latency can reflect the speed and efficiency of data updates in the target cluster. Low latency indicates that the target cluster can respond and process write operations quickly.

[0069] In this embodiment, if the data of the source cluster and the target cluster are completely consistent and the playback delay of the target cluster is within the preset delay threshold, the detection result can be determined as meeting the traffic switching conditions. If the data of the source cluster and the target cluster are inconsistent, or the playback delay of the target cluster is not within the preset delay threshold, the detection result can be determined as not meeting the traffic switching conditions.

[0070] Optionally, in the data migration method provided in this application embodiment, the traffic switching conditions include: the data of the source cluster and the updated target cluster are consistent, the playback delay of the playback operation is less than a preset threshold, and the detection result is obtained by checking whether the verification result and the playback delay meet the traffic switching conditions. The detection result includes: if the verification result indicates that the first snapshot and the second snapshot are consistent and the playback delay is less than the preset threshold, it is determined that the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions; if the verification result indicates that the first snapshot and the second snapshot are inconsistent, and / or the playback delay is not less than the preset threshold, it is determined that the detection result indicates that the source cluster and the updated target cluster do not meet the traffic switching conditions.

[0071] If the verification result indicates that the first snapshot and the second snapshot are completely consistent, and the replay delay is less than the preset delay threshold, it can be determined that the source cluster and the updated target cluster are ready to switch traffic. At this time, all read and write requests can be directed to the target cluster.

[0072] If the verification result indicates a data difference between the first and second snapshots, it suggests an error or omission during the incremental log replay process. If the replay operation latency exceeds a preset threshold, it indicates that the target cluster's current performance is insufficient to handle the production environment load. If any condition is not met, the detection result indicates that the source cluster and the updated target cluster do not meet the traffic switching conditions. In this case, if a data difference exists between the first and second snapshots, the target logs can be replayed, or the migration process can be paused to identify and resolve the issue (e.g., by identifying and resolving data inconsistencies between the source and target clusters, optimizing the target cluster's performance) until all conditions are met.

[0073] By detecting whether the cluster meets the traffic switching conditions, the reliability of the migration process can be ensured.

[0074] Optionally, in the data migration method provided in the embodiments of this application, the data migration method further includes: if the verification result indicates that the first snapshot and the second snapshot are inconsistent, replaying the write operation in the target cluster based on the target queue.

[0075] For example, if the first snapshot and the second snapshot are inconsistent, incremental log retransmission can be triggered, and the logs can be replayed from the target queue offset. Specifically, when the verification result indicates that the data in the first snapshot and the second snapshot are inconsistent, it may mean that one or more data states of the target cluster are inconsistent with those of the source cluster. In this case, measures can be taken to correct the data state of the target cluster to make it consistent with the source cluster. For example, the target logs can be replayed to rewrite the data involved in the write operations in the target logs into the target cluster.

[0076] In an alternative example, comparing the first and second snapshots can also determine which data or operations caused the inconsistency. Cyclic Redundancy Check (CRC) or other data verification algorithms can be used to pinpoint the specific differences. If the write operation causing the inconsistency is identified, the relevant incremental logs can be extracted from the target queue. It should be noted that the extracted incremental logs may have been missed, failed to process, or are out of order during previous replays. The extracted incremental logs can be resent to the target cluster for a replay operation. The target cluster can execute the write operations in the extracted incremental logs to update its data, correcting the differences with the source cluster. After the replay is complete, another snapshot of the target cluster can be obtained and compared with the snapshot of the source cluster to confirm whether data consistency has been restored.

[0077] In one alternative example, to improve efficiency and ensure fault tolerance, some additional measures can be taken, such as: (1) Resuming interrupted downloads: If a problem is encountered again during replay, the current processing status (such as the offset of the target queue) can be recorded, and replay can resume from the point of last interruption after the problem is resolved, avoiding repeated processing of logs that have already been successfully processed. (2) Multi-threaded replay: To speed up the replay process, the multi-threaded capability of the target cluster can be used to process multiple logs simultaneously, improving processing efficiency. (3) Monitoring and alarms: Throughout the replay process, the operation status and latency of the replay can be continuously monitored. Once an anomaly is detected (such as the processing latency exceeding the threshold), an alarm should be triggered immediately, which may require adjusting the replay strategy or increasing resource allocation.

[0078] Through the above process, even if problems are encountered during data migration, such as data inconsistency or replay failure, the data in the target cluster can be corrected by replaying the log operations, ensuring that the data status of the source cluster and the target cluster is completely consistent when the migration is finally completed, and ensuring the safety and reliability of the traffic switching process.

[0079] Optionally, in the data migration method provided in this application embodiment, adding the target log to the target queue includes: determining whether the data volume of the target log is greater than a preset data volume threshold, and obtaining a determination result; if the determination result indicates that the data volume of the target log is greater than the preset data volume threshold, splitting the target log into N log fragments, where N is a positive integer; and adding the N log fragments to the target queue.

[0080] In distributed systems, data migration often involves a large number of write operations, each of which has a certain size when converted into a log. To manage memory and processing load on the target cluster, a preset data volume threshold can be set. This threshold can serve as an upper limit for the log size. If the result indicates that the data volume of the target log exceeds the preset data volume threshold, the target log can be sharded. For example, if the size of a single target log exceeds 1MB, the target log can be sharded, and the ID of the log shard can be appended to the key hash to obtain N log shards. This ensures that each log shard still contains complete operation information, but the data volume is greatly reduced, making it easier to manage and replay. The fields of the operation information can include, but are not limited to: OpCode, Key, Value, and Timestamp.

[0081] Optionally, in the data migration method provided in this application embodiment, the fields of each log shard operation information in the target log may include: operation code, key value, and timestamp.

[0082] An optional target log format is shown in Table 1.

[0083] Table 1

[0084]

[0085] Example log: SET|user:123|{"name":"Alice"}|1698567123456789.

[0086] After the target log is divided into N log shards, each log shard can be added to the target queue. The target queue caches the log shards and waits for the consumer threads of the target cluster to read them from the target queue and perform replay operations.

[0087] In this embodiment, adding log shards to the target queue instead of directly adding large amounts of target logs to the target queue can improve the efficiency of data migration and thus accelerate the entire data migration process.

[0088] Optionally, in the data migration method provided in this application embodiment, based on the target queue, the write operation is replayed in the target cluster to obtain the updated target cluster, including: starting M threads to concurrently pull logs in the target queue, where M is a positive integer; in the target cluster, based on the conflict-free replication data type algorithm, the logs pulled by the M threads are replayed to obtain the updated target cluster.

[0089] During data migration, write operations from the source cluster can be transformed into a series of structured logs. These structured logs can then be added to the target queue, which acts as a buffer between the source and target clusters, ensuring that all write operations are delivered to the target cluster in a timely and orderly manner.

[0090] To improve the data synchronization process, M consumer threads can be started simultaneously to pull logs from the target queue for processing. The number of consumer threads can be dynamically adjusted based on factors such as the number of CPU cores, memory capacity, network bandwidth, and the amount of logs in the target queue.

[0091] In this embodiment, log replay can also be performed based on a conflict-free replication data type algorithm. This algorithm can handle conflicts during data synchronization in distributed systems, allowing data on different nodes to be independently replicated and updated, ensuring consistency and conflict-free results during final merging. In remote dictionary service cluster migration scenarios, the conflict-free replication data type algorithm can help handle data out-of-order or conflict issues that may arise from multi-threaded concurrent log fetching and replay. For example, the algorithm compares the timestamp or version number of each update operation to determine which operation is the latest, thus preserving the most up-to-date data state.

[0092] By using multi-threaded concurrent pull of all write operation logs from the target queue and employing a conflict-free replication data type algorithm to handle out-of-order and data conflict issues during concurrent updates, a target cluster with the same data state as the source cluster is ultimately obtained, achieving the technical effect of zero downtime and high-efficiency database cluster migration.

[0093] Optionally, in the data migration method provided in this application embodiment, before intercepting write operations on the marked data slots and converting the write operations into structured logs to obtain the target logs, the method includes: binding the target address to the source cluster, wherein the target address includes: the network address accessed by the access traffic; migrating the access traffic from the source cluster to the updated target cluster includes: binding the target address to the updated target cluster.

[0094] The target address mentioned above can include the network address accessed by the access traffic. In order to achieve a smooth transition from the source cluster to the target cluster, the target address can first be bound to the source cluster, meaning that the client will still access the data through the address of the source cluster without knowing it.

[0095] Once data migration and synchronization have progressed to a certain point and the traffic switching conditions are met, the target address can be bound to the updated target cluster. This will allow all client traffic to be rerouted to the target cluster instead of passing through the source cluster. The binding operation can be achieved by updating DNS records, modifying configuration files, or adjusting load balancer rules to ensure that client requests are correctly routed to the updated target cluster.

[0096] In database cluster migration, binding the target address to both the source and updated target clusters is a crucial step in ensuring service continuity and data consistency during the migration process. This ensures that the migration process is transparent to client applications, thereby avoiding any service interruptions caused by the migration.

[0097] In this embodiment, zero-downtime cluster migration can be achieved through structured logs and message queue pipelines. The comparison between the first and second snapshots can reduce the data error rate. By utilizing target address drift and automatic flow switching, the switching time is short. In this embodiment, second-level rollback can also be performed to improve the reliability of data migration.

[0098] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0099] Example 2

[0100] Embodiment 2 of the present invention provides an optional data migration system, which can be used to execute the data migration method provided in Embodiment 1 of the present invention.

[0101] Figure 3 This is a schematic diagram of a data migration system provided according to an embodiment of this application, such as... Figure 3 As shown, it includes: application layer, intelligent agent layer, topology system, and message queue.

[0102] 1. Intelligent Proxy Layer: Intercepts application requests and automatically handles MOVED / ASK responses; enables dual writes during migration, writing data to both the source and target clusters, and ensuring write order consistency through transaction IDs.

[0103] 2. Incremental Log Synchronization Engine: Hooks the source cluster kernel write operations and generates structured logs. The fields in the structured logs may include, but are not limited to: OpCode, Key, Value, and Timestamp. The structured logs are transmitted asynchronously via a message queue and replayed in parallel by multiple threads on the target cluster. During the parallel replay process, the problem of out-of-order collisions can be solved by a conflict-free copy data type algorithm.

[0104] 3. Dynamic Topology Coordinator: Verifies the data consistency between the source cluster and the target cluster based on the Cyclic Redundancy Check (CR) algorithm; and monitors the replay latency of the target cluster in real time (e.g., whether the replay latency is less than the replay latency threshold of 50ms) and the duration of the latency, and automatically triggers route switching based on the monitored cluster replay latency data.

[0105] The aforementioned modules intercept application requests and implement dual-write forwarding through an intelligent proxy layer; asynchronously synchronize data between the source and target clusters through an incremental log pipeline; and seamlessly trigger route switching based on consistent snapshots and replay latency thresholds through a dynamic topology coordinator.

[0106] Figure 4 This is a schematic diagram of the data migration process provided in the embodiments of this application. The following is in conjunction with... Figure 4 The data migration process will be explained as follows:

[0107] 1. Consistency Verification Phase: Every preset period (e.g., daily), the dynamic topology coordinator can calculate the first snapshot of the data slots in the source cluster based on the Cyclic Redundancy Check (CR) algorithm, and calculate the second snapshot of the data slots in the target cluster based on the CR algorithm, and check the consistency between the first snapshot of the source cluster and the second snapshot of the target cluster.

[0108] 2. Latency monitoring phase: Real-time detection of the target cluster replay latency. If the snapshots of the source cluster and the target cluster are consistent, it can detect whether the replay latency is less than the preset latency threshold and continues for a preset duration, for example, whether the replay latency is less than 50 milliseconds and continues for 5 minutes.

[0109] 3. Routing switch phase: The dynamic topology coordinator updates the virtual IP binding to the target cluster.

[0110] Disaster recovery mechanism: When consistency verification between the source and target clusters fails (e.g., the first and second snapshots are inconsistent), incremental log retransmission is triggered (replaying from the message queue offset). A 24-hour reverse synchronization channel is maintained after the switchover, supporting second-level rollback. If consistency verification between the source and target clusters passes, and the replay latency is less than a preset latency threshold for a preset duration, the virtual IP can be updated to the target cluster via the dynamic coordinator, and dual-write can be disabled.

[0111] Figure 5 This is a schematic diagram of another data migration process provided according to an embodiment of this application, such as... Figure 5 As shown, it includes:

[0112] The first phase, the initialization phase, includes the following steps:

[0113] Step 1. Mark the migration data slots:

[0114] In this embodiment, a migration command can be sent to the source cluster through a dynamic topology coordinator: marking all data slots of data to be migrated as "pending migration" status;

[0115] The proxy layer binds a virtual IP (corresponding to the target address in Example 1) to the source cluster. In an optional example, the virtual IP can be "VIP: 10.0.0.1". In this case, application traffic can access the source cluster through the virtual IP.

[0116] Step 2. Enable double write buffering:

[0117] The intelligent proxy layer can forward write requests for data slots that need to be migrated to both the source and target clusters simultaneously, and use transaction ID sequences to ensure write order consistency (e.g., TXID: timestamp + clientID).

[0118] Phase Two: Incremental Synchronization Phase, including the following steps:

[0119] Step 1. Log generation and transmission:

[0120] In this embodiment, write operations to the source cluster (e.g., SET / DEL / INCR, etc.) can be intercepted through the source cluster kernel hook, generating structured logs of the write operations to obtain the target logs. The target logs are then added to the target queue. Before adding the target logs to the target queue, the target logs can be sharded. For example, if the value of a single target log exceeds 1MB, log sharding can be performed, and the shard IDs of each log shard can be automatically appended to the key hash.

[0121] Step 2. Target cluster log replay:

[0122] To improve log replay efficiency, in this embodiment, multiple concurrent consumer threads can be started to pull target logs from the target queue and perform log replay on the target cluster. During the log replay process, out-of-order conflict issues can also be resolved by using a conflict-free copy data type algorithm.

[0123] Phase 3: Hot-swap phase, including the following steps:

[0124] Step 1. Consistency check:

[0125] The dynamic topology coordinator can generate a first snapshot and a second snapshot at a preset frequency (e.g., daily), and compare the first snapshot and the second snapshot to obtain the data differences between the source cluster and the target cluster.

[0126] If the first snapshot and the second snapshot are inconsistent, incremental log retransmission can be triggered, and log replay can be performed based on the offset of the target queue.

[0127] Step 2. Delay monitoring and switching conditions:

[0128] In this embodiment, the replay latency of the target cluster can be monitored in real time, and the cluster switching conditions can be determined based on the replay latency and consistency check results. Optionally, the switching conditions can be: the replay latency is ≤ a preset latency threshold and lasts for a preset duration, for example, the replay latency is ≤ 50ms for 5 minutes, and the first snapshot and the second snapshot are consistent.

[0129] The dynamic topology coordinator sends a virtual IP switching command to bind the virtual IP to the target cluster.

[0130] Phase Four: Cleanup Phase, including the following steps:

[0131] Step 1. Disable dual write and release resources:

[0132] The intelligent proxy layer shuts down writes to the source cluster, and 100% of application traffic is switched to the target cluster.

[0133] The source cluster deletes the migrated data slots.

[0134] Step 2. Reverse the synchronization window (24 hours):

[0135] The target cluster synchronizes logs in reverse with the source cluster, and supports second-level rollback in case of an error.

[0136] In this embodiment, zero-downtime cluster migration can be achieved through structured logs and message queue pipelines. The consistency comparison between the first and second snapshots can reduce the data error rate. By utilizing virtual IP drift and automatic flow switching, the switching time is short. In this embodiment, second-level rollback can also be performed to improve the reliability of data migration.

[0137] Example 3

[0138] This application also provides a data migration apparatus. It should be noted that the data migration apparatus of this application can be used to execute the data migration method provided in this application. The data migration apparatus provided in this application will be described below.

[0139] According to embodiments of this application, an apparatus for implementing the above-described data migration method is also provided, such as... Figure 6 As shown, the device includes: a marking unit 61, a first processing unit 62, a second processing unit 63, a detection unit 64, and a migration unit 65.

[0140] The marking unit 61 is used to mark the data slots of the data to be migrated in the source cluster in response to the data migration request, and obtain the marked data slots. The data migration request is used to request the migration of the data to be migrated in the source cluster to the target cluster.

[0141] The first processing unit 62 is used to intercept write operations on the marked data slots and convert the write operations into structured logs to obtain the target log;

[0142] The second processing unit 63 is used to add the target log to the target queue and, based on the target queue, perform a replay operation on the write operation in the target cluster to obtain the updated target cluster. The replay operation is used to write the data associated with the write operation to the target cluster.

[0143] Detection unit 64 is used to detect whether the source cluster and the updated target cluster meet the traffic switching conditions and obtain the detection result;

[0144] Migration unit 65 is used to migrate access traffic from the source cluster to the updated target cluster when the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions.

[0145] In the data migration apparatus provided in this application embodiment, the marking unit 61 responds to the data migration request by marking the data slots of the data to be migrated in the source cluster, thus obtaining marked data slots. The data migration request requests the migration of the data to be migrated from the source cluster to the target cluster. The first processing unit 62 intercepts write operations on the marked data slots and converts the write operations into structured logs, obtaining the target log. The second processing unit 63 adds the target log to the target queue and, based on the target queue, performs a replay operation on the write operations in the target cluster, obtaining an updated target cluster. The replay operation writes the data associated with the write operations to the target cluster. The detection unit 64 detects whether the source cluster and the updated target cluster meet the traffic switching conditions, obtaining a detection result. If the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions, the migration unit 65 migrates the access traffic from the source cluster to the updated target cluster. This solves the technical problem of high downtime risk when using migration tools for cluster migration in related technologies. In this embodiment, data migration is performed by bidirectional writing to the source and target clusters through message queues and incremental logs, achieving zero-downtime migration and thus improving the efficiency of cluster data migration.

[0146] Optionally, in the data migration apparatus provided in this application embodiment, the detection unit includes: a first acquisition subunit, used to acquire a data snapshot of the source cluster to obtain a first snapshot, wherein the data snapshot includes: the result obtained after processing the snapshot of the data slot based on the cyclic redundancy check algorithm; a second acquisition subunit, used to acquire an updated data snapshot of the target cluster to obtain a second snapshot; a verification subunit, used to perform consistency verification on the first snapshot and the second snapshot to obtain a verification result; a third acquisition subunit, used to acquire the replay delay of the replay operation; and a detection subunit, used to detect whether the verification result and the replay delay meet the traffic switching conditions to obtain a detection result.

[0147] Optionally, in the data migration apparatus provided in this application embodiment, the traffic switching conditions include: the data of the source cluster and the updated target cluster are consistent, and the playback delay of the playback operation is less than a preset threshold. The detection subunit includes: a first processing module, used to determine that the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions when the verification result indicates that the first snapshot and the second snapshot are consistent and the playback delay is less than the preset threshold; and a determining module, used to determine that the detection result indicates that the source cluster and the updated target cluster do not meet the traffic switching conditions when the verification result indicates that the first snapshot and the second snapshot are inconsistent, and / or when the playback delay is not less than the preset threshold.

[0148] Optionally, in the data migration apparatus provided in the embodiments of this application, the data migration apparatus further includes: a replay unit, used to replay the write operation in the target cluster based on the target queue when the verification result indicates that the first snapshot and the second snapshot are inconsistent.

[0149] Optionally, in the data migration apparatus provided in this application embodiment, the second processing unit includes: a judgment subunit, used to judge whether the data volume of the target log is greater than a preset data volume threshold, and obtain a judgment result; a sharding subunit, used to shard the target log to obtain N log shards when the judgment result indicates that the data volume of the target log is greater than the preset data volume threshold, where N is a positive integer; and an adding subunit, used to add the N log shards to the target queue.

[0150] Optionally, in the data migration apparatus provided in this application embodiment, the second processing unit includes: a startup subunit, used to start M threads to concurrently pull logs from the target queue, where M is a positive integer; and a replay subunit, used to perform a replay operation on the logs pulled by the M threads in the target cluster based on a conflict-free replication data type algorithm, to obtain an updated target cluster.

[0151] Optionally, in the data migration apparatus provided in this application embodiment, the data migration apparatus further includes: a binding subunit, used to bind the target address to the source cluster before intercepting write operations on the marked data slots and converting the write operations into structured logs to obtain the target logs, wherein the target address includes: the network address accessed by the access traffic; the migration unit includes: a binding subunit, used to bind the target address to the updated target cluster.

[0152] Optionally, in the data migration apparatus provided in this application embodiment, the fields of the target log include: operation code, key value, and timestamp.

[0153] It should be noted that the marking unit 61, the first processing unit 62, the second processing unit 63, the detection unit 64, and the migration unit 65 mentioned above correspond to steps S201 to S205 in Embodiment 1. Each unit and the corresponding step implement the same instance and application scenario, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0154] Example 4

[0155] Embodiments of this application may provide an electronic device. Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 7 As shown, the electronic device may include: one or more ( Figure 7 (Only one is shown) processor 702, memory 704, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0156] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0157] The processor can invoke information and applications stored in memory via the transmission device to perform the following steps: In response to a data migration request, mark the data slots of the data to be migrated in the source cluster to obtain the marked data slots, wherein the data migration request is used to request the migration of the data to be migrated from the source cluster to the target cluster; intercept write operations on the marked data slots and convert the write operations into structured logs to obtain the target logs; add the target logs to the target queue and, based on the target queue, perform a replay operation on the write operations in the target cluster to obtain the updated target cluster, wherein the replay operation is used to write the data associated with the write operations to the target cluster; detect whether the source cluster and the updated target cluster meet the traffic switching conditions to obtain the detection result; if the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions, migrate the access traffic from the source cluster to the updated target cluster.

[0158] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: detecting whether the source cluster and the updated target cluster meet the traffic switching conditions and obtaining the detection result, including: acquiring a data snapshot of the source cluster to obtain a first snapshot, wherein the data snapshot includes: the result obtained after processing the data slot snapshot based on the cyclic redundancy check algorithm; acquiring a data snapshot of the updated target cluster to obtain a second snapshot; performing a consistency check on the first snapshot and the second snapshot to obtain a check result; acquiring the playback delay of the playback operation; and detecting whether the check result and the playback delay meet the traffic switching conditions to obtain the detection result.

[0159] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: The traffic switching conditions include: the data of the source cluster and the updated target cluster are consistent, and the playback delay of the playback operation is less than a preset threshold. The detection result is obtained by checking whether the verification result and the playback delay meet the traffic switching conditions, including: if the verification result indicates that the first snapshot and the second snapshot are consistent and the playback delay is less than the preset threshold, it is determined that the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions; if the verification result indicates that the first snapshot and the second snapshot are inconsistent, and / or the playback delay is not less than the preset threshold, it is determined that the detection result indicates that the source cluster and the updated target cluster do not meet the traffic switching conditions.

[0160] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: The data migration method further includes: if the verification result indicates that the first snapshot and the second snapshot are inconsistent, replaying the write operation in the target cluster based on the target queue.

[0161] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: adding the target log to the target queue, including: determining whether the data volume of the target log is greater than a preset data volume threshold, and obtaining a determination result; if the determination result indicates that the data volume of the target log is greater than the preset data volume threshold, splitting the target log into N log fragments, where N is a positive integer; and adding the N log fragments to the target queue.

[0162] The processor can also invoke information and applications stored in memory via the transmission device to perform the following steps: based on the target queue, perform a replay operation on the write operation in the target cluster to obtain an updated target cluster, including: starting M threads to concurrently pull logs from the target queue, where M is a positive integer; in the target cluster, perform a replay operation on the logs pulled by the M threads based on a conflict-free copy data type algorithm to obtain an updated target cluster.

[0163] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: before intercepting write operations to the marked data slots and converting the write operations into structured logs to obtain the target log, including: binding the target address to the source cluster, wherein the target address includes: the network address accessed by the access traffic; migrating the access traffic from the source cluster to the updated target cluster, including: binding the target address to the updated target cluster.

[0164] The processor can also access information and applications stored in the memory via a transmission device to perform the following steps: The fields of the target log include: opcode, key value, and timestamp.

[0165] By employing the embodiments of this application, data migration is achieved through bidirectional writing to the source and target clusters via message queues and incremental logs, thus achieving the goal of zero-downtime migration and improving the technical efficiency of cluster data migration.

[0166] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 7 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0167] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0168] Example 5

[0169] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data migration method provided in Embodiment 1.

[0170] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0171] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing data migration method steps.

[0172] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0178] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data migration method, characterized in that, include: In response to a data migration request, the data slots of the data to be migrated in the source cluster are marked to obtain the marked data slots. The data migration request is used to request the migration of the data to be migrated in the source cluster to the target cluster. Intercept write operations to the marked data slots and convert the write operations into structured logs to obtain the target log; The target log is added to the target queue, and based on the target queue, the write operation is replayed in the target cluster to obtain the updated target cluster. The replay operation is used to write the data associated with the write operation into the target cluster. Detect whether the source cluster and the updated target cluster meet the traffic switching conditions, and obtain the detection result; If the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions, the access traffic will be migrated from the source cluster to the updated target cluster.

2. The data migration method according to claim 1, characterized in that, The detection results include: checking whether the source cluster and the updated target cluster meet the traffic switching conditions. Obtain a data snapshot of the source cluster to obtain a first snapshot, wherein the data snapshot includes: the result obtained after processing the snapshot of the data slot based on the cyclic redundancy check algorithm; Obtain a data snapshot of the updated target cluster to obtain a second snapshot; Perform a consistency check on the first snapshot and the second snapshot to obtain the check result; Obtain the playback delay of the playback operation; The detection result is obtained by checking whether the verification result and the playback delay meet the traffic switching conditions.

3. The data migration method according to claim 2, characterized in that, The traffic switching conditions include: the data of the source cluster and the updated target cluster are consistent, and the replay delay of the replay operation is less than a preset threshold. The detection result includes checking whether the verification result and the replay delay meet the traffic switching conditions. If the verification result indicates that the first snapshot and the second snapshot are consistent, and the playback delay is less than the preset threshold, then the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions. If the verification result indicates that the first snapshot and the second snapshot are inconsistent, and / or if the playback delay is not less than the preset threshold, it is determined that the detection result indicates that the source cluster and the updated target cluster do not meet the traffic switching conditions.

4. The data migration method according to claim 2, characterized in that, The method further includes: If the verification result indicates that the first snapshot and the second snapshot are inconsistent, the write operation is replayed in the target cluster based on the target queue.

5. The data migration method according to claim 1, characterized in that, Add the target log to the target queue, including: Determine whether the data volume of the target log is greater than a preset data volume threshold, and obtain the determination result; If the judgment result indicates that the data volume of the target log is greater than the preset data volume threshold, the target log is sharded to obtain N log shards, where N is a positive integer; Add N log shards to the target queue.

6. The data migration method according to claim 1, characterized in that, Based on the target queue, the write operation is replayed in the target cluster to obtain an updated target cluster, including: Start M threads to concurrently pull logs from the target queue, where M is a positive integer; In the target cluster, based on the conflict-free replication data type algorithm, the logs pulled by the M threads are replayed to obtain the updated target cluster.

7. The data migration method according to claim 1, characterized in that, Before intercepting write operations to the marked data slots and converting the write operations into structured logs to obtain the target logs, the process includes: binding the target address to the source cluster, wherein the target address includes: the network address accessed by the access traffic; Migrating access traffic from the source cluster to the updated target cluster includes: binding the target address to the updated target cluster.

8. The data migration method according to claim 1, characterized in that, The target log fields include: opcode, key value, and timestamp.

9. A data migration device, characterized in that, include: A marking unit is used to mark the data slots of the data to be migrated in the source cluster in response to a data migration request, thereby obtaining the marked data slots. The data migration request is used to request the migration of the data to be migrated in the source cluster to the target cluster. The first processing unit is used to intercept write operations to the marked data slots and convert the write operations into structured logs to obtain the target logs; The second processing unit is used to add the target log to the target queue, and based on the target queue, to perform a replay operation on the write operation in the target cluster to obtain an updated target cluster, wherein the replay operation is used to write the data associated with the write operation into the target cluster. The detection unit is used to detect whether the source cluster and the updated target cluster meet the traffic switching conditions and obtain the detection result; A migration unit is configured to migrate access traffic from the source cluster to the updated target cluster when the detection result indicates that the source cluster and the updated target cluster meet the traffic switching conditions.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the data migration method according to any one of claims 1 to 8.

Images