Data migration method and device between clusters, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data migration method and device between clusters, electronic equipment and a storage medium, and relates to the technical field of distributed storage, the data migration method between the clusters is applied to a version coordination layer, and the method comprises the steps that to-be-migrated data information of read-only cluster instances of a physical instance layer is determined, the to-be-migrated data information comprises target data, a data identifier of the target data and a current version identifier of the target data; according to the data identifier of the target data, calling a logic cluster layer to query each version identifier of the target data from a physical instance layer to obtain the latest version identifier of the target data; comparing the current version identifier and the latest version identifier of the target data to obtain a comparison result; and determining whether to migrate the target data to the read-write cluster instance of the physical instance layer or not according to a comparison result. According to the embodiment of the invention, a distributed lock can be abandoned, data migration without loss of performance is realized, a data reviving phenomenon is avoided, and data consistency is ensured.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of distributed storage, and particularly relate to a data migration method and device between clusters, an electronic device and a storage medium. BACKGROUND

[0002] In a management system of super large scale data, multiple clusters can be combined into a specific logical architecture WR (KV1, KV2) to provide services externally. Among them, KV1 and KV2 are two clusters, WR (Writable-Readable) is used to describe the read-write attribute of the cluster, KV1 is a read-write cluster, and KV2 is a read-only cluster.

[0003] Under the WR architecture, all write operations only occur on KV1, and KV2 does not receive any write request. When a deletion request involves a record existing in KV2, the corresponding records in KV1 and KV2 can be deleted at the same time. In the process of implementing the present application, the inventors found that this deletion method can introduce concurrent security problems in the data migration scenario.

[0004] That is, if a record is just read out from KV2 during the migration process and has not been written into KV1, at this time the user initiates a deletion instruction to delete the corresponding records in KV1 and KV2, and then the migration process writes the record into KV1, the deleted data will be "revived" again, causing data inconsistency. To ensure data safety during migration, the traditional method is to introduce a global distributed lock to ensure that the migration task and the deletion operation are executed mutually exclusively. However, the distributed lock not only reduces the system throughput, but also increases the architecture complexity and operation and maintenance cost. SUMMARY

[0005] Embodiments of the present application provide a data migration method and device between clusters, an electronic device and a storage medium, which can discard the distributed lock, realize lossless performance of data migration, eliminate the "revival" phenomenon of data, and ensure data consistency.

[0006] In a first aspect, the data migration method between clusters provided by the embodiments of the present application is applied to a version coordination layer, and includes:

[0007] Determine the to-be-migrated data information of the read-only cluster instance of the physical instance layer, the to-be-migrated data information including target data, a data identifier of the target data, and a current version identifier of the target data;

[0008] According to the data identifier of the target data, call the logical cluster layer to query each version identifier of the target data from the physical instance layer, and obtain the latest version identifier of the target data;

[0009] Compare the current version identifier of the target data with the latest version identifier to obtain a comparison result;

[0010] Based on the comparison results, determine whether to migrate the target data to the read-write cluster instance at the physical instance layer.

[0011] Secondly, the data migration apparatus between clusters provided in this embodiment of the invention is applied to a version coordination layer, including:

[0012] The determination module is used to determine the data information to be migrated from the read-only cluster instance at the physical instance layer. The data information to be migrated includes the target data, the data identifier of the target data, and the current version identifier of the target data.

[0013] The query module is used to query the various version identifiers of the target data from the physical instance layer based on the data identifier of the target data, and obtain the latest version identifier of the target data.

[0014] The comparison module is used to compare the current version identifier and the latest version identifier of the target data and obtain the comparison result;

[0015] The decision module is used to determine whether to migrate the target data to the read-write cluster instance at the physical instance layer based on the comparison results.

[0016] Thirdly, the electronic device provided in the embodiments of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data migration method between clusters as described in any embodiment of the present invention.

[0017] Fourthly, the computer-readable storage medium provided in the embodiments of the present invention stores a computer program thereon, which, when executed by a processor, implements the data migration method between clusters as described in any embodiment of the present invention.

[0018] In this embodiment of the invention, by comparing the current version identifier and the latest version identifier of the target data, concurrency control is transformed from coarse-grained "locks" to fine-grained "data version verification." This is a lock-free design, allowing migration operations and user deletion operations to be executed concurrently without waiting for each other, and system throughput is no longer constrained by distributed locks. Migration will only be executed when the version (current version identifier) ​​read from the read-only cluster instance is the latest version in the entire system. If the deletion operation occurs first, it will generate a higher version of data in the read-write cluster instance. At this time, the version number of the old data in the read-only cluster instance is no longer the latest version identifier, and the migration condition update will fail, thus preventing the old data from being written. If the migration operation occurs first, after the data is successfully migrated, its version is updated in the read-write cluster instance, and subsequent deletion operations will be processed based on this new version. That is, regardless of the timing of the two operations, concurrency conflicts can be automatically resolved at the system level, ensuring eventual data consistency and the correctness of the operation logic. By introducing a version coordination layer and version identifiers, the migration logic is decoupled from the specific implementation of the underlying storage. Regardless of the underlying storage format, as long as a version identifier can be provided for the data, it can be integrated into this migration framework. It is independent of any specific database feature, possessing strong versatility and scalability. The core of the entire solution lies in the comparison of data versions themselves. Without the need for centralized coordination components (such as lock services), the system architecture becomes simpler and more robust, reducing the number of components requiring maintenance, lowering potential points of failure, and eliminating the risk of the entire system migration stalling due to lock service downtime. The maintainability and reliability of the system are significantly improved. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a data migration method between clusters provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the data migration architecture between clusters provided in an embodiment of the present invention;

[0022] Figure 3 This is another flowchart illustrating the data migration method between clusters provided in this embodiment of the invention;

[0023] Figure 4a This is an example diagram of a data migration method between clusters provided in an embodiment of the present invention;

[0024] Figure 4b This is another example diagram of the data migration method between clusters provided in this embodiment of the invention;

[0025] Figure 4c This is another example diagram of the data migration method between clusters provided in the embodiments of the present invention;

[0026] Figure 5 This is a schematic diagram of a data migration device between clusters provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] In practical applications, the WR(KV1, KV2) architecture is usually formed as follows: Initially, the system contains only one read-write cluster. When this read-write cluster is full or fails, it is set as a read-only cluster KV2, and a new cluster KV1 is introduced to take over the read and write requests, thus forming a logical "one write, multiple reads" data system.

[0031] In the WR(KV1, KV2) architecture, all write operations occur only on KV1; KV2 does not receive any write requests. When a delete request involves a record existing in KV2, the system can adopt the following two processing strategies:

[0032] Strategy 1: Mark and delete strategy.

[0033] When a data deletion request is received, only a corresponding "deletion marker" record is inserted into KV1; the original data in KV2 is not deleted. While this approach is logically clear, during migration, if a piece of data is found to already exist in KV1 (whether valid data or a deletion marker), the migration of that data is skipped because the state in KV1 already covers that in KV2. However, when a large number of deletion operations are involved, a large number of deletion markers accumulate in KV1, leading to performance degradation and even timeouts in subsequent operations.

[0034] Strategy 2: Direct deletion strategy.

[0035] To improve efficiency, the corresponding records in both KV1 and KV2 can be deleted simultaneously upon receiving a deletion request. However, this approach can lead to data "revival" issues during data migration.

[0036] The data migration method proposed in this invention mainly addresses the problems caused by the "direct deletion strategy". Figure 1 This is a flowchart illustrating a data migration method between clusters provided in an embodiment of the present invention. The data migration method between clusters provided in this embodiment is applicable to scenarios involving data migration between different clusters. The data migration method between clusters can be executed by a data migration device between clusters provided in this embodiment, which can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer or server. The following embodiment illustrates this using the integration of the data migration device between clusters into an electronic device as an example.

[0037] Before introducing the method of the embodiments of the present invention, let's first introduce the data migration architecture between clusters provided by the embodiments of the present invention. For example... Figure 2 As shown, the architecture, from bottom to top, includes: a physical instance layer, a logical cluster layer, and a version coordination layer. Each layer provides higher-level abstractions and services to the layer above and hides the complexity of the layer below. In other words, this invention extends the original architecture by adding a version coordination layer. The data migration method of this invention can be implemented in the top-level version coordination layer. Wherein:

[0038] The physical instance layer, located at the bottom layer, is the final storage location for data. It consists of physically independent key-value storage cluster instances, which can include read-write cluster instance KV1 and read-only cluster instance KV2.

[0039] The logical cluster layer, located in the middle layer, is a routing and abstraction layer. It logically combines multiple cluster instances from the lower layers into a unified, scalable cluster to provide services to the outside world. Its core logic is WR(KV1, KV2), which defines the role relationships between clusters: KV1 is writable, and KV2 is read-only. The version coordination layer accesses the lower-level physical instances by calling this layer.

[0040] The version coordination layer, located at the top layer, builds upon the logical cluster layer and endows the entire system with Multi-Version Concurrency Control (MVCC) capabilities. Its core logic is MVCC(WR(KV1, KV2)), which receives the logical cluster view WR(KV1, KV2) provided by the lower layer and encapsulates it with MVCC semantics. The version coordination layer can assign a globally monotonically increasing version identifier to each write operation (e.g., insert, update) to achieve global version management across the cluster.

[0041] In other words, the physical instance layer provides storage capabilities, the logical cluster layer provides scalability and routing capabilities, and the version coordination layer provides intelligent consistency guarantees through MVCC and conditional update mechanisms, thereby enabling safe and correct data migration without the need for global locks and avoiding the data "revival" problem.

[0042] Continue reading Figure 1 The method in this embodiment can be executed by the version coordination layer. The data migration method between clusters in this embodiment may include the following steps:

[0043] Step 101: Determine the data information to be migrated for the read-only cluster instance at the physical instance layer. The data information to be migrated includes the target data, the data identifier of the target data, and the current version identifier of the target data.

[0044] The physical instance layer includes read-only cluster instances and read-write cluster instances. A read-only cluster instance refers to a cluster in the physical instance layer that is set to read-only mode, such as... Figure 2 In a KV2 database, data can only be read, not written to or updated. Its internal data is "frozen" at a certain point in time, becoming a static, historical snapshot. This snapshot serves as the source of data to be migrated, and during the migration process, it can still provide read-only services to the online system. A read-write cluster instance refers to a cluster at the physical instance layer that is configured for both read and write modes, such as... Figure 2 The KV1 in the data can not only be read, but also written or updated. It is the destination of data migration, and during the migration process, it still handles the write traffic and read requests on the line.

[0045] The data to be migrated is a data packet scanned from a read-only cluster instance (such as KV2) and prepared to be migrated to a read-write cluster instance (such as KV1). It is a collection of information containing three elements: target data, target data identifier, and target data current version identifier. Target data is the core content of the data to be migrated. It can be any data stored based on a key-value pair storage model, the values ​​in key-value pairs, and can include metadata of the stored object, user profiles, session information, content, and media information. For example, the actual content of a user profile (e.g., Xiao Wang, 30 years old) is a piece of target data. The target data identifier is the key used to identify this data. Continuing the previous example, for instance, the user ID or number corresponding to this user profile is 123; 123 is the data identifier for this data. The target data current version identifier is a marker of the current version of the data in the read-only cluster instance, which can be a monotonically increasing number or a timestamp. Continuing with the previous example, for instance, the version number bound to this user profile is 1, which means that in the read-only cluster instance, the value of the first version of the key 123 is (Xiao Wang, 30 years old).

[0046] This means that all the data that needs to be migrated can be found from a physical cluster in read-only mode (such as KV2), and three key pieces of information can be prepared for each piece of data: its ID (Key), its specific content (Value), and its version number (Version) in the current read-only cluster.

[0047] In practical implementation, the migration of data from read-only cluster instances at the physical instance layer can be performed in scenarios such as cluster decommissioning and upgrading, system scaling down to save costs, and storage space reclamation. For example, if the hardware of an old read-only cluster instance (such as KV2) is outdated or the software version is too low and needs to be phased out, its data needs to be migrated to a new read-write cluster instance (such as KV1) to safely decommission it. For example, initially, a "one-write-many-read" architecture was deployed for disaster recovery and performance. As the business grows, it is found that so many read-only clusters are not needed to support the traffic. To save resources, it is decided to reduce the number of clusters and merge the data of some read-only cluster instances before decommissioning them. For example, if the storage space of a read-only cluster instance (such as KV2) is about to be full, while the read-write cluster instance (such as KV1) still has a lot of space, data migration can be used to balance the storage load and release space in KV2. These operations can be performed without stopping system services or affecting normal online read and write requests.

[0048] Specifically, a scanning task can be initiated, sending a full table or full-range scan request to a read-only cluster instance (such as KV2). To improve efficiency, the scanning process can be parallelized. For each key scanned, its corresponding value and current version identifier (Key, Version, Value) are simultaneously read from KV2. This triplet information is then encapsulated into a "data to be migrated" unit and placed in a migration task queue, awaiting processing in subsequent steps. A full scan consumes KV2's input / output and computational resources; therefore, it can be performed during off-peak hours or its impact on online services can be controlled through rate limiting.

[0049] Step 102: Based on the data identifier of the target data, call the logical cluster layer to query the version identifiers of the target data from the physical instance layer to obtain the latest version identifier of the target data.

[0050] Each version identifier of the target data refers to the set of all historical version numbers that may exist in the entire system (including KV1 and KV2) for a given data identifier (Key). The latest version identifier of the target data is the version number with the largest value found by comparing the above set of version identifiers. It represents the latest and valid record version corresponding to the data identifier within the entire WR(KV1, KV2) system.

[0051] The version coordination layer does not interact directly with KV1 and KV2. Instead, it translates a single query into actual calls to multiple physical instances by invoking the logical cluster layer. Specifically, when the version coordination layer prepares to migrate a piece of data, it first initiates a "global status query" to the logical cluster layer. The logical cluster layer then simultaneously queries both the KV1 and KV2 physical clusters, for example, "query the version number of this key." The logical cluster layer then aggregates and reports all the collected version numbers, and the version coordination layer identifies the largest one, which is the latest version number.

[0052] The version coordination layer requires a global, unified view. If it directly queries KV1 and KV2, it would need to handle complex issues such as routing and network anomalies. By calling the logical cluster layer, the version coordination layer obtains a concise abstraction, encapsulating the complexity at a lower level. Through a single global query, it obtains the information needed to make the correct decision, providing data support for subsequent migration decisions.

[0053] Step 103: Compare the current version identifier and the latest version identifier of the target data to obtain the comparison result.

[0054] The comparison result is the output of a check to see if the two version identifiers are the same, and can be either True or False. True indicates that the two identifiers are the same, meaning that the data in KV2 is still globally up-to-date, and the migration can proceed safely. False indicates that the two identifiers are different, meaning that the data state has changed during the migration preparation period (e.g., it has been updated or deleted on KV1), and the migration needs to be abandoned to avoid data inconsistency.

[0055] Step 104: Determine whether to migrate the target data to the read-write cluster instance at the physical instance layer based on the comparison results.

[0056] The migration here is not a simple copy, but a write operation accompanied by version number upgrades or updates. For example, after writing (key, v1, value) from KV2 to KV1, it may become (key, v2, value) in KV1 to indicate that this is a new change.

[0057] If the comparison results are the same, the migration is executed, proving that the data version read from KV2 (e.g., v1) is still the latest version in the entire system, and no concurrent write operations have occurred. At this point, migrating the data to KV1 is safe, as it will not overwrite any new data and can trigger an atomic write operation to write the data to KV1. If the comparison results are different, the migration is not executed, proving that the system state has changed after reading KV2 but before the decision (e.g., data was updated or deleted on KV1), and the data in KV2 is an outdated copy. The migration task can be abandoned, the log is recorded, and the next data is processed to avoid data "resurrection."

[0058] In one specific embodiment, in response to a data query request for target data from a user terminal, the data query request includes a data identifier for the target data. Based on the data identifier of the target data, the logical cluster layer is invoked to query the various version identifiers of the target data from the physical instance layer to obtain the latest version identifier of the target data. The data query result is then sent to the user terminal, and the data query result includes the target data corresponding to the latest version identifier.

[0059] In essence, when a user (e.g., an application) needs to read a piece of data, it sends a query request to the system, containing the data identifier of the data it wants to retrieve. This query request is received by the version coordination layer. The version coordination layer doesn't blindly search for data in KV2, as it doesn't know if the data has been updated or deleted. Instead, it calls the logical cluster layer and requests a global query. The logical cluster layer simultaneously queries both the KV1 and KV2 physical instances, asking them for all version identifiers of the data they each possess. The logical cluster layer collects responses from KV1 and KV2, containing all version information for the data in both clusters. It then compiles this version information into a version list and sends it to the version coordination layer, which retrieves the latest version identifier from this list. The data corresponding to this latest version identifier is the most authoritative and valid data in the entire system at that moment. It may originate from either KV1 or KV2. After obtaining the data corresponding to the latest version identifier, the version coordination layer encapsulates it in a response and sends it to the user. The user receives the most accurate data available at that moment.

[0060] Users don't need to worry about which cluster the data is stored in; the system automatically provides them with the latest data view. Global version comparison ensures that read operations always yield consistent results, regardless of the intermediate state of the data during migration. This read mechanism shares the same MVCC version control core with the migration and write mechanisms described earlier, resulting in a clear system architecture and consistent behavior.

[0061] In practice, the roles of read-only cluster instances and read-write cluster instances are not static; their configuration is typically a result of system lifecycle and operational strategies. For example, initially, the system has only one cluster, a read-write cluster instance. As business grows, read requests become a bottleneck. At this point, operations personnel can deploy a new cluster (e.g., KV1) and set the old cluster (e.g., KV2) as a read-only cluster instance. All new write operations are handled by KV1, while KV2 serves the massive read requests. The architecture then becomes WR(KV1, KV2). Now, we want to take the old KV2 offline to save costs, or the hardware of KV2 needs to be phased out. We can then safely migrate all data from KV2 (the read-only cluster instance) to KV1 (the read-write cluster instance). After the migration, KV2 is taken offline, the system architecture reverts to a single-cluster WR(KV1), and KV1 continues to serve as a read-write cluster instance. This cycle can continue. In the future, when KV1 becomes a bottleneck, KV3 can be introduced as a new read-write cluster instance, and KV1 can be downgraded to a read-only cluster instance.

[0062] In this embodiment, by comparing the current version identifier and the latest version identifier of the target data, concurrency control is transformed from coarse-grained "locks" to fine-grained "data version verification." This is a lock-free design, allowing migration and user deletion operations to execute completely concurrently without waiting for each other, and system throughput is no longer constrained by distributed locks. Migration will only execute when the version (current version identifier) ​​read from the read-only cluster instance is the latest version in the entire system. If the deletion operation occurs first, it will generate a higher version of data in the read-write cluster instance. At this point, the version number of the old data in the read-only cluster instance is no longer the latest version identifier, and the migration condition update will fail, thus preventing the old data from being written. If the migration operation occurs first, after the data is successfully migrated, its version is updated in the read-write cluster instance, and subsequent deletion operations will be processed based on this new version. That is, regardless of the timing of the two operations, concurrency conflicts can be automatically resolved at the system level, ensuring eventual data consistency and the correctness of the operation logic. By introducing a version coordination layer and version identifiers, the migration logic is decoupled from the specific implementation of the underlying storage. Regardless of the underlying storage format, as long as a version identifier can be provided for the data, it can be integrated into this migration framework. It is independent of any specific database feature, possessing strong versatility and scalability. The core of the entire solution lies in the comparison of data versions themselves. Without the need for centralized coordination components (such as lock services), the system architecture becomes simpler and more robust, reducing the number of components requiring maintenance, lowering potential points of failure, and eliminating the risk of the entire system migration stalling due to lock service downtime. The maintainability and reliability of the system are significantly improved.

[0063] The following examples further illustrate the data migration method between clusters provided in this embodiment of the invention, such as... Figure 3 As shown, the method in this embodiment includes:

[0064] Step 201: Determine the data information to be migrated for the read-only cluster instance at the physical instance layer. The data information to be migrated includes the target data, the data identifier of the target data, and the current version identifier of the target data.

[0065] The target data is stored in key-value pairs. The data identifier of the target data is the key in the key-value pair, and the target data is the value in the key-value pair. The current version identifier of the target data is a marker of the current version of the data in the read-only cluster instance, which can be a monotonically increasing number or a timestamp.

[0066] Step 202: Based on the data identifier of the target data, call the logical cluster layer to query the various version identifiers of the target data from the physical instance layer to obtain the latest version identifier of the target data.

[0067] The physical instance layer includes read-only cluster instances and read-write cluster instances. Read-only cluster instances and read-write cluster instances can be homogeneous or heterogeneous. Homogeneous means that the read-only cluster instances and read-write cluster instances are the same type of key-value storage software or system. For example, KV1 and KV2 are both MySQL clusters of relational database management systems. Heterogeneous means that the read-only cluster instances and read-write cluster instances are different types of key-value storage software or systems. For example, KV1 is a MySQL cluster, and KV2 is a TiKV cluster.

[0068] The version coordination layer can send query commands to the logical cluster layer, requesting a query for the version identifier of a specified key (i.e., the key in the data to be migrated). Upon receiving this query command, the logical cluster layer, knowing its current logical architecture is WR(KV1, KV2), needs to query both the read-write cluster instance KV1 and the read-only cluster instance KV2 to obtain the global state. KV1 and KV2 execute the query within their respective clusters, finding all unreclaimed version records based on the specified key and returning them to the logical cluster layer. The logical cluster layer then compiles a list of version identifiers and returns it to the version coordination layer, which retrieves the latest version identifier for the specified key from this list.

[0069] Specifically, the version coordination layer can divide the target data into multiple shards based on the key; start multiple migration executors and assign multiple shards to multiple migration executors; use multiple migration executors to call the logical cluster layer to query the version identifiers of the corresponding shard data from the physical instance layer to obtain the latest version identifier of the corresponding shard data.

[0070] Sharding involves dividing all keys to be migrated into multiple smaller, non-overlapping subsets of data according to certain rules. Each subset is called a shard, facilitating parallel processing. Specific sharding methods include range sharding and hash sharding. Range sharding, for example, divides the data according to the lexicographical range of the keys; for instance, shard 1 has a range of [af], and shard 2 has a range of [gm]. Hash sharding, on the other hand, performs a hash calculation on the keys and allocates them to different shards based on the modulo of the hash value. This method better ensures a more even distribution of data.

[0071] A migration executor is an independent worker process or thread, representing the smallest unit of work that actually performs data migration tasks. Each migration executor can be assigned one or more shards, and it is responsible for completing the "query-migration" process for all data within those shards. Multiple executors can run concurrently without interfering with each other.

[0072] This involves dividing all the data that needs to be migrated globally into multiple logical data blocks (shards) based on their keys. Then, multiple migration executors are launched, each responsible for processing one or more data blocks. Each migration executor calls the system's query interface to retrieve version information for each piece of data in its assigned block. Multiple executors scan and migrate data simultaneously, making full use of system resources and effectively improving the overall migration speed.

[0073] Step 203: Compare the current version identifier and the latest version identifier of the target data to obtain the comparison result.

[0074] The comparison results include identical or different results.

[0075] Step 204: Determine whether the current version identifier and the latest version identifier are the same. If they are the same, proceed to step 205; otherwise, proceed to step 208.

[0076] Step 205: Determine the target data to be migrated to the read / write cluster instance at the physical instance layer.

[0077] Step 206: Call the logical cluster layer to write the target data to the read-write cluster instance of the physical instance layer, and add a new version identifier to the target data written to the read-write cluster instance.

[0078] This means that instead of simply copying the old version of data from the read-only cluster instance to the read-write cluster instance unchanged, it treats it as a new data change. A new version number, larger than the current latest version number, can be obtained from the global version number generator and written to the read-write cluster instance along with the data. Assume the data version read from KV2 is v1, and the current latest version of the system is v1 (indicating no other updates). When writing to KV1, the system will generate v2 and write (key, v2, value).

[0079] By elevating data migration to a versioned write event rather than a simple data copy, and incorporating the system's MVCC concurrency control model, eventual data consistency and high system availability are achieved without the need for global locks.

[0080] Step 207: In response to a data modification request from the user, which includes the data identifier of the target data, the logical cluster layer is invoked to modify the target data written in the read-write cluster instance of the physical instance layer, and an updated version identifier is added to the modified target data.

[0081] When the version coordination layer receives a data modification request from a user, it does not directly modify the underlying database. Instead, it uses the intermediate logical cluster layer to securely route the modification operation to a read-write cluster instance (KV1) for execution. Furthermore, during the modification process, a new, larger version number is assigned to the modified data, thus officially archiving the user operation in the system's history.

[0082] Write traffic is strictly directed to the read-write cluster instance KV1, maintaining a simple system architecture and a clear consistency model. Each modification generates a new version, creating an immutable, linear operation history for the system. It is this mechanism of generating a new version with every modification that makes lock-free migration possible in this invention. When checking versions, the migration operation automatically abandons the process if it finds that a user operation has already created an updated version, thus avoiding data conflicts.

[0083] Step 208: Determine not to migrate the target data to the read / write cluster instance at the physical instance layer.

[0084] This means abandoning the operation of writing data from KV2 to KV1. This non-migration action is just as important as performing the migration, and perhaps even more important in demonstrating the security of this invention. It is not inaction, but a proactive and safe abort action taken to protect data consistency and up-to-date state. Determining not to migrate the target data to the read-write cluster instance at the physical instance layer does not describe a failure state, but rather declares the successful implementation of a security protection measure.

[0085] In this embodiment, by comparing the current version identifier and the latest version identifier of the target data, concurrency control is transformed from coarse-grained "locks" to fine-grained "data version verification." This is a lock-free design, allowing migration and user deletion operations to execute completely concurrently without waiting for each other, and system throughput is no longer constrained by distributed locks. Migration will only execute when the version (current version identifier) ​​read from the read-only cluster instance is the latest version in the entire system. If the deletion operation occurs first, it will generate a higher version of data in the read-write cluster instance. At this point, the version number of the old data in the read-only cluster instance is no longer the latest version identifier, and the migration condition update will fail, thus preventing the old data from being written. If the migration operation occurs first, after the data is successfully migrated, its version is updated in the read-write cluster instance, and subsequent deletion operations will be processed based on this new version. That is, regardless of the timing of the two operations, concurrency conflicts can be automatically resolved at the system level, ensuring eventual data consistency and the correctness of the operation logic. By introducing a version coordination layer and version identifiers, the migration logic is decoupled from the specific implementation of the underlying storage. Regardless of the underlying storage format, as long as a version identifier can be provided for the data, it can be integrated into this migration framework. It is independent of any specific database feature, possessing strong versatility and scalability. The core of the entire solution lies in the comparison of data versions themselves. Without the need for centralized coordination components (such as lock services), the system architecture becomes simpler and more robust, reducing the number of components requiring maintenance, lowering potential points of failure, and eliminating the risk of the entire system migration stalling due to lock service downtime. The maintainability and reliability of the system are significantly improved.

[0086] The following is combined Figure 4a , Figure 4b and Figure 4c To illustrate the data migration method between clusters provided in this embodiment of the invention, suppose we want to migrate (key1, value1) from KV2 to KV1.

[0087] like Figure 4aAs shown, assume that KV1 has no record for key1, while KV2 contains (key1, version 1, value1), meaning key1 exists in KV2 but has never been manipulated in KV1 (neither written nor deleted). The migration executor reads (key1, version 1, value1) from KV2 and queries the system for the latest version of key1. The system finds that there is no record for key1 in KV1, and version 1 in KV2 is the current highest version. Since the current version (1) is equal to the latest version (1), the condition is met. The system performs a condition update, writes the data to KV1, and assigns it a new, larger version number (e.g., version 2). Therefore, (key1, version 2, value1) will eventually be generated in KV1. The migration is successful, and the data (key1, value1) will be safely migrated from KV2 to KV1.

[0088] like Figure 4b As shown, KV1 originally contained (key1, version 2, value1), and KV2 originally contained (key1, version 1, value1). During the migration process, a deletion instruction for key1 was received, and both (key1, version 2, value1) originally stored in KV1 and (key1, version 1, value1) originally stored in KV2 were deleted. That is, the migration and deletion were concurrent, and currently there is no record of key1 in either KV1 or KV2. During the migration operation, the migration executor read the old (key1, version 1, value1) from KV2. It queries the system for the latest version of key1, and the system finds that version 2 in KV1 is the highest version. Since the current version (1) is not equal to the latest version (2), the condition is not met, and this migration is abandoned. That is, the old data in KV2 will not be migrated to KV1, preventing the "revival" of deleted data.

[0089] like Figure 4c As shown, KV1 contains (key1, version 2, value1), and KV2 contains (key1, version 1, value1). This situation may occur because the migration has been completed or the data in key1 has been modified according to user instructions. The migration executor reads (key1, version 1, value1) from KV2 and queries the system for the latest version of key1. The system finds that version 2 exists in KV1, meaning the latest version is identified as version 2. Since the current version (1) is not equal to the latest version (2), the condition is not met, and the migration is abandoned. That is, the old data (version 1) in KV2 will not be migrated to KV1, preventing data rollback.

[0090] Figure 5This is a schematic diagram of a data migration device between clusters provided in an embodiment of the present invention. This device is suitable for executing the data migration method between clusters provided in an embodiment of the present invention, such as... Figure 5 As shown, the device may specifically include:

[0091] The determination module 501 is used to determine the data information to be migrated for the read-only cluster instance at the physical instance layer. The data information to be migrated includes the target data, the data identifier of the target data, and the current version identifier of the target data.

[0092] The query module 502 is used to call the logical cluster layer to query the various version identifiers of the target data from the physical instance layer based on the data identifier of the target data, so as to obtain the latest version identifier of the target data;

[0093] Comparison module 503 is used to compare the current version identifier and the latest version identifier of the target data to obtain the comparison result;

[0094] Decision module 504 is used to determine whether to migrate the target data to the read-write cluster instance at the physical instance layer based on the comparison results.

[0095] In one embodiment, the decision module 504 determines whether to migrate the target data to a read-write cluster instance at the physical instance layer based on the comparison result, including:

[0096] If the comparison result shows that the current version identifier and the latest version identifier of the target data are the same, it is determined that the target data will be migrated to the read-write cluster instance at the physical instance layer;

[0097] If the comparison result shows that the current version identifier and the latest version identifier of the target data are different, it is determined that the target data will not be migrated to the read-write cluster instance at the physical instance layer.

[0098] In one embodiment, after determining that the target data should be migrated to a read-write cluster instance at the physical instance layer, the decision module 504 is used to:

[0099] The logical cluster layer is invoked to write the target data to the read-write cluster instance of the physical instance layer, and a new version identifier is added to the target data written to the read-write cluster instance.

[0100] In one embodiment, the decision module 504 is further configured to:

[0101] In response to a data modification request from the user client for target data, which includes the data identifier of the target data, the logical cluster layer is invoked to modify the target data written in the read-write cluster instance of the physical instance layer, and an updated version identifier is added to the modified target data.

[0102] In one embodiment, the target data is stored in the form of key-value pairs, where the data identifier of the target data is the key in the key-value pair, and the target data is the value in the key-value pair.

[0103] In one embodiment, the query module 502, based on the data identifier of the target data, calls the logical cluster layer to query the various version identifiers of the target data from the physical instance layer to obtain the latest version identifier of the target data, including:

[0104] The target data is divided into multiple fragments based on the key;

[0105] Start multiple migration executors and assign multiple shards to multiple migration executors;

[0106] By using multiple migration executors to call the logical cluster layer to query the version identifiers of the corresponding shard data from the physical instance layer, the latest version identifier of the corresponding shard data can be obtained.

[0107] In one embodiment, the decision module 504 is further configured to:

[0108] In response to a data query request from the user client for target data, which includes the data identifier of the target data, the logical cluster layer is invoked to query the various version identifiers of the target data from the physical instance layer to obtain the latest version identifier of the target data.

[0109] Send the data query results to the user's end. The data query results include the target data corresponding to the latest version identifier.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] In this embodiment, by comparing the current version identifier and the latest version identifier of the target data, concurrency control is transformed from coarse-grained "locks" to fine-grained "data version verification." This is a lock-free design, allowing migration and user deletion operations to execute completely concurrently without waiting for each other, and system throughput is no longer constrained by distributed locks. Migration will only execute when the version (current version identifier) ​​read from the read-only cluster instance is the latest version in the entire system. If the deletion operation occurs first, it will generate a higher version of data in the read-write cluster instance. At this point, the version number of the old data in the read-only cluster instance is no longer the latest version identifier, and the migration condition update will fail, thus preventing the old data from being written. If the migration operation occurs first, after the data is successfully migrated, its version is updated in the read-write cluster instance, and subsequent deletion operations will be processed based on this new version. That is, regardless of the timing of the two operations, concurrency conflicts can be automatically resolved at the system level, ensuring eventual data consistency and the correctness of the operation logic. By introducing a version coordination layer and version identifiers, the migration logic is decoupled from the specific implementation of the underlying storage. Regardless of the underlying storage format, as long as a version identifier can be provided for the data, it can be integrated into this migration framework. It is independent of any specific database feature, possessing strong versatility and scalability. The core of the entire solution lies in the comparison of data versions themselves. Without the need for centralized coordination components (such as lock services), the system architecture becomes simpler and more robust, reducing the number of components requiring maintenance, lowering potential points of failure, and eliminating the risk of the entire system migration stalling due to lock service downtime. The maintainability and reliability of the system are significantly improved.

[0112] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data migration method between clusters provided in any of the above embodiments.

[0113] This invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data migration method between clusters provided in any of the above embodiments.

[0114] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device according to embodiments of the present invention. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0115] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0116] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0117] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0118] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] The modules and / or units described in the embodiments of the present invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor can be described as including a determining module, a querying module, a comparison module, and a decision module. The names of these modules do not necessarily limit the module itself.

[0121] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0122] The process involves determining the data to be migrated from the read-only cluster instance at the physical instance layer. This data includes the target data, the target data's data identifier, and the target data's current version identifier. Based on the target data's data identifier, the logical cluster layer is invoked to query the various version identifiers of the target data from the physical instance layer to obtain the latest version identifier. The current version identifier and the latest version identifier of the target data are compared to obtain a comparison result. Based on the comparison result, it is determined whether to migrate the target data to the read-write cluster instance at the physical instance layer.

[0123] The technical solution of this invention transforms concurrency control from coarse-grained "locks" to fine-grained "data version verification" by comparing the current version identifier and the latest version identifier of the target data. This is a lock-free design, allowing migration operations and user deletion operations to be executed concurrently without waiting for each other, and system throughput is no longer constrained by distributed locks. Migration will only be executed when the version (current version identifier) ​​read from the read-only cluster instance is the latest version in the entire system. If the deletion operation occurs first, it will generate a higher version of data in the read-write cluster instance. At this time, the version number of the old data in the read-only cluster instance is no longer the latest version identifier, and the migration condition update will fail, thus preventing the old data from being written. If the migration operation occurs first, after the data is successfully migrated, its version is updated in the read-write cluster instance, and subsequent deletion operations will be processed based on this new version. That is, regardless of the timing of the two operations, concurrency conflicts can be automatically resolved at the system level, ensuring eventual data consistency and the correctness of operation logic. By introducing a version coordination layer and version identifiers, the migration logic is decoupled from the specific implementation of the underlying storage. Regardless of the underlying storage format, as long as a version identifier can be provided for the data, it can be integrated into this migration framework. It is independent of any specific database feature, possessing strong versatility and scalability. The core of the entire solution lies in the comparison of data versions themselves. Without the need for centralized coordination components (such as lock services), the system architecture becomes simpler and more robust, reducing the number of components requiring maintenance, lowering potential points of failure, and eliminating the risk of the entire system migration stalling due to lock service downtime. The maintainability and reliability of the system are significantly improved.

[0124] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0125] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0126] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for data migration between clusters, the method comprising: The method is applied to a version coordination layer, and comprises: determining to-be-migrated data information of a read-only cluster instance of a physical instance layer, the to-be-migrated data information comprising target data, a data identifier of the target data, and a current version identifier of the target data; according to the data identifier of the target data, calling a logical cluster layer to query, from the physical instance layer, version identifiers of the target data, to obtain a latest version identifier of the target data; comparing the current version identifier of the target data with the latest version identifier, to obtain a comparison result; determining, according to the comparison result, whether to migrate the target data to a read-write cluster instance of the physical instance layer.

2. The method of claim 1, wherein, The method of determining, according to the comparison result, whether to migrate the target data to the read-write cluster instance of the physical instance layer comprises: in a case where the comparison result is that the current version identifier of the target data is the same as the latest version identifier, determining to migrate the target data to the read-write cluster instance of the physical instance layer; in a case where the comparison result is that the current version identifier of the target data is different from the latest version identifier, determining not to migrate the target data to the read-write cluster instance of the physical instance layer.

3. The method of claim 2, wherein, After determining to migrate the target data to the read-write cluster instance of the physical instance layer, the method further comprises: calling the logical cluster layer to write the target data into the read-write cluster instance of the physical instance layer, and adding a new version identifier to the target data written into the read-write cluster instance.

4. The method of claim 3, wherein, The method further comprises: in response to a data modification request for the target data from a user end, calling the logical cluster layer to modify the target data written into the read-write cluster instance of the physical instance layer, and adding an updated version identifier to the modified target data.

5. The method of claim 1, wherein, The target data is stored in the form of a key-value pair, the data identifier of the target data is a key in the key-value pair, and the target data is a value in the key-value pair.

6. The method of claim 5, wherein, According to the data identifier of the target data, calling the logical cluster layer to query, from the physical instance layer, version identifiers of the target data, to obtain a latest version identifier of the target data, comprises: dividing the target data into a plurality of shards according to keys; starting a plurality of migration executors, and allocating the plurality of shards to the plurality of migration executors; calling the logical cluster layer to query, from the physical instance layer, version identifiers of corresponding shard data by using the plurality of migration executors, to obtain a latest version identifier of the corresponding shard data.

7. The method of claim 1, wherein, The method further comprises: in response to a data query request for the target data from a user end, according to the data identifier of the target data, calling the logical cluster layer to query, from the physical instance layer, version identifiers of the target data, to obtain a latest version identifier of the target data; sending, to the user end, a data query result, the data query result comprising the target data corresponding to the latest version identifier.

8. An apparatus for data migration between clusters, the apparatus comprising: The method is applied to a version coordination layer, and comprises: a determining module configured to determine to-be-migrated data information of a read-only cluster instance of a physical instance layer, the to-be-migrated data information comprising target data, a data identifier of the target data, and a current version identifier of the target data; a querying module configured to, according to the data identifier of the target data, call a logical cluster layer to query, from the physical instance layer, version identifiers of the target data, to obtain a latest version identifier of the target data; a comparing module configured to compare the current version identifier of the target data with the latest version identifier, to obtain a comparison result; and A decision module is configured to determine whether to migrate the target data to the read-write cluster instance of the physical instance layer according to the comparison result.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the data migration method between clusters as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the data migration method between clusters as claimed in any one of claims 1 to 7. The program is executed by the processor to implement the data migration method between clusters as claimed in any one of claims 1 to 7.

Images