Data migration method and device, program product, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a data migration method and device, a program product, a storage medium and electronic equipment, and the method comprises the steps: obtaining slot position distribution information of each slot position in a fragment; determining a mapping relation table of the target slot position by using the slot position distribution information, the mapping relation table comprising a mapping relation between the emigrated data and the target slot position; determining a migration starting position of the to-be-migrated data in the fragments based on the mapping relation table; and migrating the to-be-migrated data based on the migration starting position. Through the data migration method and device, the problem that data migration cannot be effectively carried out in related technologies is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and specifically, to a data migration method and device, a program product, a storage medium, and an electronic device. Background Art

[0002] With the development of Internet technology and the application of technologies such as cloud computing and big data, the storage and processing of massive amounts of data have become the norm. To achieve high data availability and scalability, data needs to be migrated across nodes or clusters.

[0003] During data migration, it's often necessary to scan the source cluster to identify the data to be migrated. However, in related technologies, data migration tools typically migrate data from the source cluster to the target cluster through either full or incremental migrations. However, full migrations result in a large amount of synchronized data and a long migration time, while incremental migrations can only migrate backwards from the current point in time, resulting in data loss.

[0004] There is currently no effective solution to the aforementioned problem of being unable to effectively migrate data. Summary of the Invention

[0005] The embodiments of the present application provide a data migration method and apparatus, a program product, a storage medium, and an electronic device to at least solve the problem in the related art that data migration cannot be effectively performed.

[0006] According to one embodiment of the present application, a data migration method is provided, including: obtaining slot distribution information of each slot in a shard; using the slot distribution information to determine a mapping relationship table of a target slot, wherein the mapping relationship table includes a mapping relationship between migrated data and the target slot; determining a migration starting position of the data to be migrated in the shard based on the mapping relationship table; and migrating the data to be migrated based on the migration starting position.

[0007] In an exemplary embodiment, obtaining slot distribution information of each slot in a shard includes: during the process of migrating data in the shard through a data migration process, if it is determined that an abnormality occurs in the data migration process, obtaining slot distribution information of each of the slots.

[0008] In an exemplary embodiment, the slot distribution information is used to determine a mapping relationship table of the target slot, including: obtaining a mapping relationship of the target slot according to the correspondence between the target slot included in the slot distribution information and the data storage location in the shard, wherein the mapping relationship includes the storage location of the migrated data corresponding to the target slot in the shard, a slot key value, and a mapping relationship between the target slot, and the slot key value is used to mark the mapping relationship between the storage location and the target slot; and generating the mapping relationship table using the mapping relationship.

[0009] In an exemplary embodiment, before obtaining the mapping relationship of the target slot according to the correspondence between the target slot included in the target slot distribution information and the data storage location in the shard, the method further includes: determining the mapping relationship through the following steps: in the process of migrating the target data in the shard through a data migration process, determining the target storage location of the target data in the shard, wherein the target data is the data in the migrated data; obtaining a slot key value, wherein the slot key value is used to mark the mapping relationship between the target slot and the target storage location; constructing the relationship between the target storage location, the target slot and the slot key value to obtain the mapping relationship.

[0010] In an exemplary embodiment, determining the migration starting position of the data to be migrated in the shard based on the mapping relationship table includes: extracting a position mark value from the mapping relationship table, wherein the position mark value is used to mark the serial number of the storage position of the migrated data in the shard; determining the storage position matching the target position mark value as the target migration position, wherein the target migration position is the position of the last migrated data in the migrated data in the shard, and the target position mark value is a value in the position mark value that meets a preset condition; and determining the migration starting position using the target migration position.

[0011] In an exemplary embodiment, migrating the data to be migrated based on the migration starting position includes: constructing a data migration command when the data to be migrated is read from the migration starting position, wherein the data migration command includes the data to be migrated and a mark value of the to-be-migrated position of the data to be migrated; and executing the data migration command through a data migration process to migrate the data to be migrated.

[0012] In an exemplary embodiment, before using the slot distribution information to determine the mapping relationship table of the target slot, the method also includes: determining the slot code of each of the slots; setting the slot key value of each of the slot codes using preset characters, wherein the slot key value is used to mark the mapping relationship between the storage location of the data migrated from the shard in the shard and the slot.

[0013] According to another embodiment of the present application, a data migration device is provided, including: a first acquisition module, used to obtain slot distribution information of each slot in a shard; a first determination module, used to determine a mapping relationship table of a target slot using the slot distribution information, wherein the mapping relationship table includes a mapping relationship between migrated data and the target slot; a second determination module, used to determine a migration starting position of the data to be migrated in the shard based on the mapping relationship table; and a first migration module, used to migrate the data to be migrated based on the migration starting position.

[0014] According to another embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps of any of the above method embodiments when executed by a processor.

[0015] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0016] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] Through the present application, since the mapping relationship table of the target slot is determined by obtaining the slot distribution information of each slot in the shard, and the mapping relationship between the migrated data and the target slot position is recorded in the mapping relationship table, the data can be associated with the specific storage location of the target cluster, and the migrated data can be accurately tracked. When it is necessary to determine the starting migration position of the unmigrated data in the shard, the last position of the data migration can be found by checking the mapping relationship of the data in the mapping relationship table, and this can be used as the new migration starting position. In this way, after the data migration is interrupted, it can be accurately restored from the breakpoint without having to start the full migration from the beginning, thereby improving the efficiency and reliability of the data migration process. Therefore, the problem of the inability to effectively perform data migration in the related technology can be solved, and the effect of improving the reliability of data migration can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1This is a hardware structure block diagram of a server device for a data migration method according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a data migration method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the structure between slots and shards in a cluster according to an embodiment of the present application;

[0021] Figure 4 This is the process of resuming the breakpoint according to the embodiment of the present application Figure 1 ;

[0022] Figure 5 This is the process of resuming the breakpoint according to the embodiment of the present application Figure 2 ;

[0023] Figure 6 It is a structural block diagram of a data migration device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure diagram of a server device of a data migration method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 for communication functions and an input and output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data migration method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to a server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by a communication provider of the server device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a data migration method is provided. Figure 2 is a flow chart of a data migration method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0030] Step S202: Obtain slot distribution information of each slot in the shard;

[0031] Optionally, the shard in this embodiment is a shard in the source cluster. For example, the source cluster contains 3 shards (nodes). The shard may be a shard that needs to resume data migration after a data migration exception occurs during the data migration process.

[0032] Optionally, slots are a way for clusters to manage data in a distributed manner. Each slot corresponds to a portion of data. By performing a hash calculation based on the slot identification ID, the specific shard of the data in the cluster can be located. For example, the slots allocated to the shard include N, where N represents the number of slots managed by the shard. It can be any positive integer, depending on the slot distribution and design of the target cluster. The larger the value of N, the wider the range of data that the shard is responsible for. Slot distribution information refers to information on how each slot in the source cluster is distributed on the shard. For example, in a Redis cluster, data is distributed according to 16,384 slots, and each slot can be configured on a different shard. The slot distribution information of the N target slots includes, but is not limited to, slot number, data type, data size, and storage location. The slots in this embodiment are used to locate the storage location of the data stored in the shard.

[0033] Step S204: Determine a mapping relationship table of a target slot using the slot distribution information, wherein the mapping relationship table includes a mapping relationship between the migrated data and the target slot;

[0034] Optionally, migrated data refers to data that has been successfully migrated from shards in the source cluster to the target cluster during the data migration process.

[0035] Optionally, a mapping table is a key data structure that records the relationship between each target slot in the target cluster and the data migration status of the source cluster, including the most recently migrated data location (typically represented by an offset). This table can be considered a "snapshot" of the data migration. This mapping table allows accurate tracking of the data migration completion status for each target slot, even if there are interruptions during the data migration process, thereby determining where to resume the next migration.

[0036] Step S206: determining a migration starting position of the data to be migrated in the shard based on the mapping relationship table;

[0037] Optionally, the migration starting position is the starting point for continued migration during a data migration task, particularly when the migration process is interrupted and restarted. This position is typically determined based on the location of the last successfully migrated data, particularly the migration breakpoint, and is recorded in the mapping table as an offset. In the resume mechanism, the migration starting position determines which data needs to be transferred again and which data has been successfully migrated, thereby avoiding duplicate migrations and omissions, and ensuring the accuracy and efficiency of data migration.

[0038] Optionally, data to be migrated refers to data sets in the shards that have not yet been transferred to the target cluster. During the data migration process, due to various possible interruptions, some data may have been successfully migrated while other data may still remain in the source cluster waiting to be transferred.

[0039] Step S208: Migrate the data to be migrated based on the migration starting position.

[0040] For example, in a specific embodiment, assume there is a source Redis cluster with 3 shards (nodes), each with 16384 / 3≈5461 slots. Now, the data of one of the shards needs to be migrated to a new target cluster with 5 shards (nodes). Before the migration begins, the shards are assigned N=1000 target slots, which are randomly selected from the total 16384 slots to ensure even data distribution.

[0041] According to the above example, the first step is to obtain the distribution information of the 1000 target slots in the target cluster, that is, to determine which target slots are allocated to this shard. Secondly, based on these target slots, an initial empty mapping table is established to record the latest offset position of subsequently migrated data in the target slots. Whenever data is migrated to a slot in the shard, the offset value of the corresponding slot in the mapping table is updated to the latest data position.

[0042] If an interruption occurs during data migration (such as a network failure or tool restart), the latest offset information for the target slot can be immediately read from the mapping table to determine the data migration location before the interruption. From this location, the remaining data can be migrated until all data has been migrated to the target cluster.

[0043] Among them, the execution entity of the above steps can be a terminal, a server, a specific processor set in the terminal or server, or a processor or processing device set relatively independently from the terminal or server, etc., but is not limited to this, for example, a processor or server set in a data migration tool.

[0044] Optionally, the data migration method of this embodiment can be applied to scenarios where data migration or data synchronization is required, especially in scenarios with large data volumes and high business continuity requirements. For example, scenarios where Redis cluster data migration and synchronization are required to be efficient and reliable. Redis cluster is an open source, high-performance key-value storage database that provides a variety of data structures to store data, such as strings, hashes, lists, sets, ordered sets, etc. Redis cluster stores data in memory to provide fast read and write access speeds and can persist data to disk asynchronously. It supports replication, Lua scripts, transaction processing, different levels of persistence options, and interfaces for multiple client languages. Redis cluster is widely used in caching, message queues, short-term data storage, and high-performance application scenarios. Redis cluster includes Redis shards, that is, a single master node in the Redis cluster. Redis cluster also includes slots, that is, fixed slots in the Redis cluster, totaling 16,384, and each Redis shard will have a partial slot range. Data migration and synchronization between instances in a Redis cluster can be accomplished using the RedisShake tool, an open-source tool developed based on Redis-port. RedisShake supports data transfer across various architectures, including standalone and clustered instances. It supports online real-time synchronization and offline file import across versions and architectures (standalone to standalone, standalone to cluster, and cluster to cluster).

[0045] Through the above steps, since the mapping relationship table of the target slot is determined by obtaining the slot distribution information of each slot in the shard, and the mapping relationship between the migrated data and the target slot position is recorded in the mapping relationship table, the data can be associated with the specific storage location of the target cluster, and the migrated data can be accurately tracked. When it is necessary to determine the starting migration position of the unmigrated data in the shard, the last position of the data migration can be found by checking the mapping relationship of the data in the mapping relationship table, and this can be used as the new migration starting position. In this way, after the data migration is interrupted, it can be accurately restored from the breakpoint without having to start the full migration from the beginning, thereby improving the efficiency and reliability of the data migration process. Therefore, the problem of the inability to effectively perform data migration in the related technology can be solved, and the effect of improving the reliability of data migration can be achieved.

[0046] In an exemplary embodiment, in step S202, the slot distribution information of each slot in the shard is obtained, including: in the process of migrating data in the shard through the data migration process, when it is determined that the data migration process is abnormal, the slot distribution information of each slot is obtained.

[0047] Optionally, abnormalities in the data migration process in this embodiment include, but are not limited to, interruptions in large-scale data migration tasks due to factors such as network fluctuations, hardware failures, software vulnerabilities, or human error. In such cases, obtaining shard slot distribution information, particularly offset information regarding migrated data, becomes crucial for the smooth operation of the retransmission mechanism. Offset information typically refers to the most recent location of data transmission, which helps the system determine where to resume data migration after the interruption point, thereby avoiding data loss or duplicate transmission.

[0048] For example, in a Redis cluster, if an unexpected event causes Redisshake to restart, Redisshake will not be able to know the last synchronized offset position. It is necessary to obtain the slot distribution information to determine the mapping relationship table of N target slots, and then determine the migration starting position of the data to be migrated in the shard based on the mapping relationship table to continue migrating the data.

[0049] When the data migration process encounters an abnormal interruption, this embodiment can effectively determine the breakpoint resumption location of data migration by obtaining and utilizing the slot distribution information of the shards, ensure the integrity and accuracy of data migration, avoid unnecessary data duplication and omissions, and improve the efficiency and robustness of data migration.

[0050] In an exemplary embodiment, in step S204, the slot distribution information is used to determine a mapping relationship table of the target slot, including: obtaining a mapping relationship of the target slot according to the correspondence between the target slot included in the slot distribution information and the data storage location in the shard, wherein the mapping relationship includes the storage location of the migrated data corresponding to the target slot in the shard, the slot key value and the mapping relationship between the target slot, and the slot key value is used to mark the mapping relationship between the storage location and the target slot; and generating the mapping relationship table using the mapping relationship.

[0051] Optionally, in this embodiment, the mapping relationship records the actual storage location of the data, the slot key value, and the relationship between the slots when the data is migrated from the shard in the source cluster to the slot of the shard in the target cluster. The slot key value is a special mark used to identify and track the slot to which a specific data item belongs, ensuring the accuracy and consistency of data migration. That is, the slot key value slotKey is the key of the offset of a single slot in the source cluster stored in the cluster. The generation rules of slotKey include but are not limited to: redis-shake-{slotIndex}-XXX, slotIndex is used to represent the slot subscript, XXX is used to represent the preset characters used as filler (for example, 1~9, a~Z); ensure that slotKey is eventually routed to the slot corresponding to slotIndex crc64(slotKey)=slotIndex. 16384 slotKeys can be uniformly generated in advance and remain fixed.

[0052] Optionally, the mapping table in this embodiment integrates the mapping relationships of N target slots. Each entry in the table contains the storage location corresponding to the target slot, the slot key value, and the data migration status (e.g., the offset value). The use of the mapping table helps track and manage the data migration status of each slot during the data migration process. In particular, when the data needs to be resumed after an interruption, it provides a clear identification of the breakpoint location, allowing the migration to continue from the interruption point, avoiding duplicate migration or missing data.

[0053] For example, there are 2 slots in the shard (in actual applications, the value of slot N of the shard can be any value between 16384). The slot distribution of the shard is slot_1 and slot_2. The data that has been migrated through slot_1 of the shard is Value_1, and the data that has been migrated through slot_2 is Value_2 (Value_1 and Value_2 are only for a brief introduction to the embodiment of the present application, and the amount of data migrated in the actual scenario is large). Assume that the position of Value_1 is Offset_1, and the position of Value_2 is Offset_2. The slot key value of Value_1 is SlotKey_1, and the position of Value_2 is SlotKey_2. The mapping relationship table includes the target mapping relationship between SlotKey_1 and Offset_1 of slot_1, and the mapping relationship between SlotKey_2 and Offset_2 of slot_2. In this embodiment, the offset is recorded for each slot of the shard.

[0054] This embodiment uses slot distribution information to determine the target slot mapping and generate a mapping table, a key step in enabling data migration to resume after interruption. The mapping table not only accurately tracks the status of data migration but also provides the starting point information required for resuming data transfers in the event of an interruption, ensuring the integrity and efficiency of the migration process.

[0055] In an exemplary embodiment, before obtaining the mapping relationship of the target slots according to the correspondence between the target slots and the data storage locations in the shards included in the target slot distribution information, the method further includes: determining the mapping relationship by the following steps:

[0056] Step S302, in the process of migrating target data in the shard through a data migration process, determining a target storage location of the target data in the shard, wherein the target data is data in the migrated data;

[0057] Step S304: obtaining a slot key value, wherein the slot key value is used to mark a mapping relationship between the target slot and the target storage location;

[0058] Step S306: construct a relationship among the target storage location, the target slot, and the slot key value to obtain the mapping relationship.

[0059] Optionally, the data migration process in this embodiment is a software program or service for moving data from one system (source cluster) to another system (target cluster). For example, RedisShake, as a data migration process, can not only handle batch data migration between Redis instances, but also achieve real-time data synchronization, including breakpoint resumption. It migrates or synchronizes data from the source Redis cluster to the target Redis cluster by parsing the incremental logs on the source side or backing up files using persistent storage, and ensures the accuracy and efficiency of data migration through slot mapping and offset value management.

[0060] Optionally, during data migration, a target mapping is a data structure that describes the relationship between data storage locations, slots, and slot keys within a shard. This ensures that data is accurately placed in the correct slots and identifies this mapping relationship with slot keys, facilitating subsequent data query and management, as well as resuming data migration after interruptions.

[0061] Optionally, data is stored at a specific physical or logical location on the shard, which is determined by the target slot distribution information. The data storage location record provides the exact location information of the data on the shard.

[0062] Optionally, during data migration to the target cluster, each data item is mapped to a specific target slot, called the primary target slot. To identify the mapping between data and slots, a primary slot key is introduced. This unique identifier tracks the slot to which data belongs. This key allows for rapid location of the data item after a data migration or interruption, enabling data retransmission or verification.

[0063] For example, suppose you are migrating data from shard S1 in source cluster S to shard T1 in target cluster T. Shard T1 is responsible for slots 3277-6553 of 16384 slots. The target data D is determined to be stored at physical address P1 in shard T1 during the data migration process, belonging to the first target slot 7500. The detailed steps include:

[0064] S11, Determine the storage location and first target slot of target data D: During the data migration process, the system detects that data D will be migrated to physical address P1, slot 7500 on shard T1, i.e., the first target slot. This location information is determined using slot distribution information and the synchronization mechanism during the data migration process.

[0065] S12, Obtaining the First Slot Key: To identify the mapping between data D and slot 7500, a first slot key is generated or obtained. Assume it is redis-shake-7500-XXX. This key is generated based on the slot distribution information and an algorithm (such as CRC64), ensuring the unique correspondence between the slot key and the slot.

[0066] S13, Build Target Mapping: Based on the above information, a target mapping is constructed, recording the relationship between data D's storage location P1, its first target slot 7500, and the first slot key redis-shake-7500-XXX. This mapping can be stored in the internal state of the data migration process or in the metadata of the target cluster T for subsequent data management and recovery after an outage.

[0067] S14, Application of the target mapping relationship under the resume mechanism: Assume that data migration is interrupted when migrating data D from slot 7500 to offset = 12000. Using the target mapping relationship, data D's storage location P1 and its first target slot 7500 can be quickly located. Based on the information of offset = 12000 in slot 7500, data migration can be resumed from the interruption point, avoiding duplicate data migration or possible data loss, and ensuring the continuity and integrity of data migration.

[0068] S15, Exception Handling and Recovery: During the data migration process, if an exception is detected, such as data D failing to migrate to position P1 of shard T1, the data migration process will perform exception handling and data recovery operations based on the established target mapping relationship. This may include re-migrating data D to the correct location, updating the offset value of the first target slot 7500, and ensuring that the records in the mapping relationship table are consistent with the actual data status.

[0069] This embodiment determines the storage location of target data during data migration, uses slot key values ​​to mark the mapping between data and slots, and constructs a target mapping to support data migration tracking and resume data after abnormal interruptions. This not only improves data migration efficiency but also enhances its robustness and data integrity.

[0070] In an exemplary embodiment, in step S206, the migration starting position of the data to be migrated in the shard is determined based on the mapping relationship table, including: extracting a position mark value from the mapping relationship table, wherein the position mark value is used to mark the serial number of the storage location of the migrated data in the shard; determining the storage location that matches the target position mark value as the target migration position, wherein the target migration position is the position of the last migrated data in the migrated data in the shard, and the target position mark value is a value in the position mark value that meets a preset condition; and using the target migration position to determine the migration starting position.

[0071] Optionally, in this embodiment, a series of location marker values ​​are used in the mapping table to track the data's storage location in the target cluster. These marker values ​​can be the data's offset within the target cluster and represent the sequence number of the last location reached by the migrated data in each target slot, i.e., the migration endpoint. Using these location marker values, the data migration status of each slot can be quickly determined, including whether the migration is complete and where it was interrupted.

[0072] Optionally, in this embodiment, when determining the migration starting location, the preset condition refers to a rule or standard for selecting or determining the location marker value that is most suitable for the migration starting location. For example, a possible preset condition may be selecting the highest offset value to ensure that data migration continues from the interruption point and avoid data duplication.

[0073] For example, suppose you are migrating data from shard S1 in source cluster S to shard T1 in target cluster T. Shard T1 covers slots 6554-9830 in the target cluster. During the migration, for some reason, the data migration is interrupted at slot 7000. The offset value before the interruption is recorded as 100000, which is the target location marker value. You now need to resume the data migration from the interruption point, that is, determine the migration starting point to continue the data migration. The specific steps are as follows:

[0074] S11, extracting location tag values: Extract the offset values ​​of all slots from the mapping table, i.e., N location tag values. Each location tag value represents the storage location number of the data in S1, i.e., the migration destination.

[0075] S12, Filter Target Location Marker Value: Filter or determine the location marker value that is most suitable for the migration starting point based on preset conditions. Assuming the preset condition is to select the highest offset value as the interruption point, the offset value of slot 7000, 100000, is the target location marker value.

[0076] S13, determining the migration starting position: taking the filtered target position mark value 100000 as the migration starting position, that is, data migration will start from the next position of the interruption point offset=100000, that is, offset=100001.

[0077] The process of determining the migration starting position from the mapping relationship table in this embodiment is the core of the data migration breakpoint resumption mechanism. It allows the data migration process to quickly locate and resume the data migration status before the interruption after any interruption, thereby improving the efficiency and robustness of data migration.

[0078] In an exemplary embodiment, in step S208, migrating the data to be migrated based on the migration starting position includes: constructing a data migration command when the data to be migrated is read from the migration starting position, wherein the data migration command includes the data to be migrated and a mark value of the migration position of the data to be migrated; and executing the data migration command through a data migration process to migrate the data to be migrated.

[0079] Optionally, the data migration command in this embodiment is a special instruction used to instruct the target cluster how to handle data migration. The command contains the specific content of the data to be migrated and the target location mark value. The target location mark value is used to update the migration status of the corresponding slot in the target cluster to ensure the continuity and consistency of data migration. When constructing the data migration command, it is necessary to ensure the integrity and consistency of the data. This may involve organizing the data into commands in the Redis protocol format, for example, a transaction execution command, which contains the actual synchronization data command and the mapping relationship from slotKey to the new offset, and is accompanied by a target location mark value to indicate the data status update in the target cluster.

[0080] Optionally, the target location tag value is a sequence number recorded in the mapping table that identifies the last location of the data in the shard. When resuming data migration, the data migration process updates the slot status of the shard with the new location tag value (usually the new offset value after the migration command is executed).

[0081] Optionally, when handling large-scale data migration, the data migration process may need to support concurrent migration, that is, processing data migration for multiple slots at the same time. In addition, strategies such as data compression and data transmission optimization can further improve the efficiency of data migration.

[0082] For example, suppose you are migrating data from shard S1 of source cluster S to shard T1 of target cluster T, and you have determined that the migration start position for slot 7000 is offset = 100001. Now, you are ready to start migrating the remaining data in slot 7000 from this point. The specific implementation steps include:

[0083] S11, read the data to be migrated: starting from the migration starting position offset = 100001, the data migration process reads the data to be migrated in slot 7000 of the source shard S1.

[0084] S12, Build Data Migration Command: For each batch of data read, the data migration process builds a corresponding data migration command. The command contains not only the specific data content but also the target location tag value 100001, which the target cluster uses to update the mapping table when processing the data migration.

[0085] S13, execute the data migration command: The data migration process sends the constructed data migration command to shard T1. After receiving the command, shard T1 executes the data migration, stores the data to the correct location of the corresponding slot, and updates the mapping relationship of the slot, that is, updates the target location mark value to the new offset value. For example, offset = 100500, indicating that the data migration of slot 7000 in shard T1 has reached the new location.

[0086] S14, Data Migration with Resumable Transfer Mechanism: During the data migration process, if an interruption occurs, such as a network problem causing data migration to be suspended, the data migration process can resume data migration based on the target location marker value recorded in the mapping table. For example, after the interruption is recovered, slot 7000 will resume data migration from offset = 100501 (i.e., the position after the target location marker value after the previous interruption) until all pending data is migrated.

[0087] This embodiment's process of migrating data based on its starting location plays a key role in the data migration resumability mechanism. Through meticulous data processing and status marking, it ensures the continuity, integrity, and efficiency of data migration while reducing the cost and risk of data migration caused by system interruptions or failures.

[0088] In an exemplary embodiment, before determining the mapping relationship table of the target slot using the slot distribution information, the method further includes:

[0089] Step S402, determining the slot code of each slot;

[0090] Step S404: setting a slot key value of each slot code using preset characters, wherein the slot key value is used to mark the mapping relationship between the storage location of the data migrated from the shard in the shard and the slot.

[0091] Optionally, in a Redis Cluster environment, data is distributed based on slots, of which there are 16,384 slots. The slot key generation rule is a naming convention used to create a key name that uniquely identifies each slot offset. For example, the rule can be defined as redis-shake-{slotIndex}-XXX, where {slotIndex} is the index of the slot, and XXX can be a series of padding characters (1-9, a-Z) to prevent key name conflicts and ensure a one-to-one correspondence between slots and keys.

[0092] Optionally, during data migration or synchronization, the offset is the current position of the data read cursor, recording the data point reached during the last data migration or synchronization. By storing the offset as the value of a specific key (here, SlotKey), this key can be read during the next operation to determine the starting point for migrating or synchronizing data, avoiding duplicate data migration and potential data loss. To ensure recovery after a data migration tool restart or system failure, the SlotKey and its corresponding offset value should be persistently stored, perhaps in the shards of the target cluster. During data migration or synchronization, only the offset values ​​corresponding to the SlotKey of slots where data has changed need to be updated. This reduces metadata management overhead, especially in large-scale data migration scenarios. According to the SlotKey generation rule, crc64(slotKey) = slotIndex, this ensures that the SlotKey can directly correspond to the slot after calculation using the hash function, thereby reducing the time complexity of data location and access.

[0093] Optionally, in the context of a data migration tool or service, a configuration file is used to store operational parameters, rules, and other metadata. For Redis data migration, the configuration file may contain a pre-generated list of slot keys, information about the source and target clusters, migration rules, and more. Storing slot keys in the configuration file facilitates reading these slot keys during data migration tool initialization to establish a mapping between slots and offsets.

[0094] For example, Figure 3 As shown in the figure, 16,384 SlotKeys are pre-generated through the SlotKey generation rules, and then remain fixed and used as the key for offset storage. The SlotKey value is stored in the configuration file. The SlotKey generation rules are designed to create a set of key names that can uniquely identify the 16,384 slots. These key names should meet the following requirements: Uniqueness: Each SlotKey corresponds to a slot, ensuring a one-to-one correspondence between the slot and the SlotKey to prevent data confusion. Predictability: The generation of SlotKey should follow certain rules so that its SlotKey can be predicted at any time based on the slot index. CRC64 Hash rule: After the SlotKey is calculated using the CRC64 Hash algorithm, the result should be equal to the index value of the slot. This ensures a direct correspondence between the SlotKey and the slot through the Hash algorithm, making it easy to quickly locate the distribution of slots in the cluster.

[0095] The process of pre-generating SlotKeys includes: generating a SlotKey for each slot according to the SlotKey generation rules, for example, redis-shake-{slotIndex}-XXX, where XXX can be a random filler character to ensure the uniqueness of the SlotKey. The 16,384 generated SlotKeys are stored in the configuration file of the data migration tool. This step ensures that the mapping relationship between SlotKeys and slots can be quickly restored even after the data migration tool is restarted or a fault is recovered. During the data migration or synchronization process, whenever the data status of a slot changes, such as when the data is migrated to a certain offset, the corresponding SlotKey is used to update the offset value as a mark of the current data migration status.

[0096] This embodiment uses SlotKey as the key for offset storage, which is an important part of the breakpoint-resume strategy during data migration. It not only simplifies the data migration tool's management of slot data migration status, but also ensures the continuity and integrity of data migration.

[0097] The following describes the above method using a specific example, migrating data in a Redis cluster. If RedisShake, the data migration tool, experiences an unexpected restart during the Redis cluster migration, RedisShake needs to have a breakpoint-resume feature to ensure data integrity and accuracy.

[0098] like Figure 4 As shown in the figure, RedisShake integrates the breakpoint resume function. The breakpoint resume function is implemented by the following steps:

[0099] S401 determines the SlotKey-offset mapping relationship. This specifically includes starting with a pre-generated SlotKey list stored in the configuration file. For each SlotKey, the corresponding offset value is read from the target cluster. This may require querying or directly accessing the database state of the target cluster. A key-value mapping relationship is established between each read SlotKey and its corresponding offset value to prepare for the subsequent resumable transfer strategy.

[0100] S402: Load the SlotKey-offset mapping relationship into RedisShake, specifically including:

[0101] Integration during the startup phase: During the initial startup of the RedisShake tool, ensure that the SlotKey-offset mapping relationship established in S401 is loaded into RedisShake's internal state or memory.

[0102] Aggregate mapping: This involves merging the SlotKey-offset mappings read from multiple target shards into a global, unified mapping that covers all slots and offset states, making it easier for tools to manage them uniformly.

[0103] S403: Calculate the maximum offset of the SlotKey list corresponding to the shard, specifically including:

[0104] Filter related SlotKeys: Based on the slot distribution of the current source cluster shard, filter all SlotKeys related to the shard from the full SlotKey-offset mapping relationship.

[0105] Calculate the maximum offset: Traverse the mapping relationship of these related SlotKeys and find the maximum value among all offset values. This maximum value reflects the farthest point of the shard data transmission before the interruption, that is, the starting point of data migration.

[0106] S404: Query the data of a single slot for synchronization, specifically including:

[0107] Locating the slot: Based on the maximum offset of the shard calculated in S403, locate the first slot that needs to be synchronized.

[0108] Read data: Read new data from the slot of the source cluster since the last known offset. This may involve using the Redis PSYNC command to traverse the specified slot data.

[0109] Data migration: Send the read data to the target cluster in the original key-value pair format and perform data migration operations, such as using SET or HSET commands.

[0110] S405: Match the shards in the target cluster by slot and construct a synchronization command, which specifically includes:

[0111] Slot matching: Use the slot distribution information of the target cluster to match the read data with the shards of the corresponding slots in the target cluster to determine the shards to which the data should be written.

[0112] Construct commands: Based on data synchronization requirements, construct corresponding Redis commands, such as SET and HSET, and add commands to update the SlotKey-offset mapping relationship (for example, use the SET command to update the new offset value corresponding to the SlotKey stored in the target cluster).

[0113] Transaction encapsulation: Encapsulate the above data synchronization command and SlotKey-offset update command into one transaction to ensure the atomicity of data synchronization and status update, that is, either both succeed or both fail.

[0114] S406: Save the synchronized data and the offset value corresponding to the SlotKey, including:

[0115] Execute transaction: Submit the transaction constructed in S405, synchronize the data to the target cluster, and update the offset value of the SlotKey stored in the target cluster to reflect the latest data migration status.

[0116] State persistence: After successful synchronization, the updated SlotKey-offset mapping relationship is persistently stored. It can be updated in the configuration file or in the persistence mechanism of the target cluster (such as AOF, RDB) for use in the next data synchronization or fault recovery.

[0117] Memory update: Updates the SlotKey-offset mapping relationship in RedisShake's memory to ensure that the current status of the tool is consistent with the actual migration status, facilitating subsequent operations.

[0118] This embodiment ensures the consistency and efficiency of data migration through this series of detailed steps. In particular, it enables recovery from the correct breakpoint after an interruption or failure. It also maintains transaction atomicity during data synchronization, improving the reliability and integrity of data migration. These steps are closely linked and together form a robust breakpoint-resumeable data migration framework.

[0119] In another specific embodiment, the offset of the data synchronized by the source cluster is stored in the corresponding shard of the target cluster using the mapping relationship between SlotKey and offset, and the maximum offset in the corresponding mapping relationship is taken through the slotKey list of the single shard of the source cluster to record and calculate the offset of the synchronized data, thereby achieving the purpose of breakpoint resuming. The specific process is as follows Figure 5 As shown, the following steps are included:

[0120] S501: Generate a SlotKey for each slot according to the SlotKey rule (e.g., redis-shake-{slotIndex}-XXX), ensuring the uniqueness of each SlotKey. For example, 16,384 SlotKeys are generated and stored in a local file as a fixed mapping table so that they can be quickly loaded and used when the tool is started or restarted.

[0121] S502: When the tool starts, all slot keys are loaded from the local file. A CRC-64 check is performed on each slot key to ensure that they comply with the slot mapping rules. Incorrect slot keys are ignored or marked as errors. After the check passes, the current offset values ​​corresponding to these slot keys are loaded. These values ​​reflect the progress of data migration.

[0122] S503: Read all pre-generated SlotKeys and their corresponding offset values ​​from the target cluster to form a set of SlotKey-offset mapping relationships. This is usually done through Redis's HGETALL or similar commands.

[0123] S504: Update the SlotKey-offset mapping set in the source cluster based on the latest offset value obtained in the target cluster to ensure that the mapping relationship between the two clusters is synchronized;

[0124] S505: Create a reader for each shard or node in the source cluster to read the data of the shard.

[0125] S506: Allocate an independent thread to each reader to ensure that the data of each shard can be read and synchronized in parallel;

[0126] S507: Each reader identifies the slotIndex set it is responsible for, which is based on the slot distribution configuration of the source cluster.

[0127] S508, the reader batch reads the SlotKey-offset set corresponding to the slotIndex set it is responsible for to obtain the latest offset information;

[0128] S509: For each reader, calculate the maximum offset from the SlotKey-offset set obtained by it, and use it as the starting point for data synchronization of the reader;

[0129] At step S510, the reader starts reading data from its slot slotIndex based on the calculated offset, preparing to synchronize it to the target cluster.

[0130] S511: After obtaining the writer, the writer of the shard to which the data should be written is determined based on the data's slotIndex and the slot distribution of the target cluster. A transaction is constructed that includes data synchronization commands (such as SET and HSET) and commands to update the SlotKey-offset mapping to ensure the atomicity of data synchronization and status updates. That is, all operations either succeed or fail.

[0131] S512: Execute the transaction constructed in the previous step, synchronize the data to the target cluster, and update the offset value corresponding to the SlotKey in the target cluster;

[0132] S513: Update the SlotKey-offset mapping set in the memory or local file to reflect the latest offset status.

[0133] S514: Asynchronously write the new offset information back to the source cluster, update its status, and ensure that the mapping relationship between the source cluster and the target cluster remains consistent.

[0134] S515: If any exceptions are encountered during data synchronization, such as network problems, data inconsistencies, or tool failures, the migration process is immediately terminated. The exception information is recorded and the administrator or developer is notified for intervention. After the problem is fixed, data synchronization can be resumed from the breakpoint.

[0135] The above steps in this embodiment ensure the efficiency, consistency, and resumable nature of the data migration or synchronization process. By using independent reader threads, SlotKey-offset mappings, and a transaction execution mechanism, the reliability and performance of data migration are improved. Furthermore, exception handling is incorporated to enhance the robustness of the system, ensuring that even in the event of a failure, manual intervention can be used to repair the problem and resume the data migration process.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0137] This embodiment also provides a data migration device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0138] Figure 6 is a structural block diagram of a data migration device according to an embodiment of the present application, such as Figure 6 As shown, the device includes:

[0139] A first acquisition module 62 is configured to acquire slot distribution information of each slot in a shard;

[0140] A first determining module 64 is configured to determine a mapping relationship table of a target slot using the slot distribution information, wherein the mapping relationship table includes a mapping relationship between the migrated data and the target slot;

[0141] A second determining module 66 is configured to determine a migration starting position of the data to be migrated in the shard based on the mapping relationship table;

[0142] The first migration module 68 is configured to migrate the data to be migrated based on the migration starting position.

[0143] In an exemplary embodiment, the above-mentioned first acquisition module 62 includes: a first determination unit, which is used to obtain slot distribution information of each slot when it is determined that an abnormality occurs in the data migration process during the data migration of the data in the shard through the data migration process.

[0144] In an exemplary embodiment, the above-mentioned first determination module 64 includes: a first acquisition unit, used to obtain the mapping relationship of the target slot according to the correspondence between the target slot included in the slot distribution information and the data storage location in the shard, wherein the mapping relationship includes the storage location of the migrated data corresponding to the target slot in the shard, the slot key value and the mapping relationship between the target slot, and the slot key value is used to mark the mapping relationship between the storage location and the target slot; a first generation unit, used to generate the mapping relationship table using the mapping relationship.

[0145] In an exemplary embodiment, the above-mentioned device also includes: a third determination module, which is used to determine the target mapping relationship through the following steps before obtaining the mapping relationship of the target slot according to the correspondence between the target slot included in the target slot distribution information and the data storage location in the shard: in the process of migrating the target data in the shard through the data migration process, determine the target storage location of the target data in the shard, wherein the target data is the data in the migrated data; obtain a slot key value, wherein the slot key value is used to mark the mapping relationship between the target slot and the target storage location; construct a relationship between the target storage location, the target slot and the slot key value to obtain the mapping relationship.

[0146] In an exemplary embodiment, the second determination module 66 includes: a first extraction unit for extracting a position tag value from the mapping relationship table, wherein the position tag value is used to mark the serial number of the storage location of the migrated data in the shard; a second determination unit for determining the storage location that matches the target position tag value as the target migration location, wherein the target migration location is the location of the last migrated data in the migrated data in the shard, and the target position tag value is a value in the position tag value that meets a preset condition; and a fourth determination unit for determining the migration starting position using the target migration position.

[0147] In an exemplary embodiment, the first migration module 68 includes: a first construction unit, used to construct a data migration command when the data to be migrated is read from the migration starting position, wherein the data migration command includes the data to be migrated and the mark value of the position to be migrated of the data to be migrated; a first execution unit, used to execute the data migration command through a data migration process to migrate the data to be migrated.

[0148] In an exemplary embodiment, the above-mentioned device also includes: a fourth determination module, used to determine the slot code of each of the slots before using the slot distribution information to determine the mapping relationship table of the target slot; a setting module, used to set the slot key value of each of the slot codes using preset characters, wherein the slot key value is used to mark the mapping relationship between the storage location of the data migrated from the shard in the shard and the slot.

[0149] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0150] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0151] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0152] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0153] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0154] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0155] An embodiment of the present application further provides another computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0156] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above method embodiments.

[0157] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0158] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0159] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A data migration method, characterized in that: include: Get the slot distribution information of each slot in the shard; Determine a mapping relationship table of a target slot using the slot distribution information, wherein the mapping relationship table includes a mapping relationship between the migrated data and the target slot; Determine a migration starting position of the data to be migrated in the shard based on the mapping relationship table; The data to be migrated is migrated based on the migration starting position.

2. The method according to claim 1, characterized in that Get the slot distribution information for each slot in the shard, including: In the process of migrating the data in the shard through the data migration process, if it is determined that an abnormality occurs in the data migration process, the slot distribution information of each of the slots is obtained.

3. The method according to claim 1, characterized in that Determining a mapping relationship table of target slots using the slot distribution information includes: Obtaining a mapping relationship for the target slot according to a correspondence between the target slot and a data storage location in the shard included in the slot distribution information, wherein the mapping relationship includes a mapping relationship between a storage location in the shard of the migrated data corresponding to the target slot, a slot key value, and the target slot, wherein the slot key value is used to mark the mapping relationship between the storage location and the target slot; The mapping relationship table is generated using the mapping relationship.

4. The method according to claim 3, characterized in that Before acquiring the mapping relationship of the target slots according to the corresponding relationship between the target slots and the data storage locations in the shards included in the target slot distribution information, the method further includes: The mapping relationship is determined by the following steps: In a process of migrating target data in the shard through a data migration process, determining a target storage location of the target data in the shard, wherein the target data is data in the migrated data; Obtaining a slot key value, wherein the slot key value is used to mark a mapping relationship between the target slot and the target storage location; A relationship between the target storage location, the target slot, and the slot key value is constructed to obtain the mapping relationship.

5. The method according to claim 1, wherein Determining a migration starting position of the data to be migrated in the shard based on the mapping relationship table includes: Extracting a location tag value from the mapping relationship table, wherein the location tag value is used to mark the sequence number of the storage location of the migrated data in the shard; Determine a storage location that matches a target location tag value as a target migration location, wherein the target migration location is the location of the last migrated data in the migrated-out data in the shard, and the target location tag value is a value among the location tag values ​​that meets a preset condition; The migration starting position is determined using the target migration position.

6. The method according to claim 5, characterized in that Migrating the data to be migrated based on the migration starting position includes: In a case where the data to be migrated is read from the migration starting position, a data migration command is constructed, wherein the data migration command includes the data to be migrated and a mark value of a to-be-migrated position of the data to be migrated; The data migration command is executed through a data migration process to migrate the data to be migrated.

7. The method according to claim 1, characterized in that Before determining the mapping relationship table of the target slot using the slot distribution information, the method further includes: Determining a slot code for each of the slots; A slot key value of each slot code is set using preset characters, wherein the slot key value is used to mark the mapping relationship between the storage location of the data migrated from the shard in the shard and the slot.

8. A data migration device, characterized in that: include: A first acquisition module is used to obtain slot distribution information of each slot in the shard; a first determining module, configured to determine a mapping relationship table of a target slot using the slot distribution information, wherein the mapping relationship table includes a mapping relationship between the migrated data and the target slot; A second determining module is configured to determine a migration starting position of the data to be migrated in the shard based on the mapping relationship table; A first migration module is configured to migrate the data to be migrated based on the migration starting position.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images