Data migration methods, components, clients, electronic devices, and readable storage media

Abstract

This disclosure provides a data migration method, component, client, electronic device, and storage medium, relating to the fields of computer and internet technology. The method includes: when a source node synchronizes incremental instructions to a target node, a data migration component intercepts and temporarily stores instructions sent by a client to the source node, wherein the incremental instructions are instructions received by the source node from the client after generating a snapshot file in response to the synchronization instructions, and the snapshot file is a snapshot generated by the source node based on existing data in the source node after responding to the synchronization instructions; the data migration component sends a probe key to the source node so that the source node continues to synchronize the probe key after synchronizing the incremental instructions to the target node; the data migration component detects the probe key at the target node according to a target period; when the data migration component detects the probe key at the target node, the data migration component sends the intercepted and temporarily stored instructions sent by the client to the target node so that the target node continues to provide services to the client.

Description

Technical Field

[0001] This disclosure relates to the field of computer and Internet technology, and in particular to a data migration method, component, client, electronic device and computer-readable storage medium. Background Technology

[0002] As technology advances, the scale of cluster nodes continues to grow, and the deployment methods of cluster nodes (such as Redis cluster nodes) are also changing, from early direct deployment on physical machines to current deployment on managed services based on cloud computing platforms. Furthermore, because the cluster solutions provided by the open-source versions of cluster nodes are not perfect, cloud vendors develop their own cluster modes based on their specific needs in their managed services. This results in a situation where, although everyone is using cluster nodes (such as Redis cluster nodes), their architectures, versions, and other aspects differ. Therefore, during data migration, it is not possible to simply rely on master-slave replication mechanisms to migrate data between heterogeneous nodes; instead, cross-node data synchronization solutions are required for data migration.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0004] The purpose of this disclosure is to provide a data migration method, component, client, electronic device, and computer-readable storage medium, providing a cross-node data migration solution.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] This disclosure provides a data migration method, comprising: when a source node synchronizes incremental instructions to a target node, a data migration component intercepts and temporarily stores instructions sent by a client to the source node, wherein the source node is used to provide services to the client, the incremental instructions are instructions received by the source node from the client after generating a snapshot file in response to a synchronization instruction, the snapshot file is a snapshot generated by the source node based on existing data in the source node in response to the synchronization instruction, wherein the incremental instructions are synchronized to the target node later than the snapshot file; the data migration component sends a probe key to the source node so that the source node continues to synchronize the probe key after synchronizing the incremental instructions to the target node; the data migration component detects the probe key at the target node according to a target period; when the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node; the data migration component sends the intercepted and temporarily stored instructions sent by the client to the target node so that services can continue to be provided to the client through the target node.

[0007] In some embodiments, before the source node synchronizes incremental instructions to the target node, the method further includes: when the source node synchronizes the snapshot file to the target node, the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node; wherein, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node, including: the data migration component intercepts and temporarily stores the instructions sent by the client to the source node; the data migration component does not forward the instructions sent by the client to the source node.

[0008] In some embodiments, the source node synchronizes the probe key to the target node via a synchronization tool. A period of time after the probe key synchronization is complete, the source node generates and sends probe key failure information, which is used to delete the probe key. Furthermore, after the data migration component determines that the source node has completed data synchronization with the target node, the method further includes: the data migration component sending a stop signal to the synchronization tool so that the synchronization tool no longer synchronizes the data sent by the source node to the target node, thereby preventing the source node from sending the probe key failure information to the target node.

[0009] In some embodiments, the data migration component includes a control subcomponent and a proxy subcomponent; wherein, when the source node synchronizes incremental instructions to the target node, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node, including: when the source node synchronizes the incremental instructions to the target node, the control subcomponent sends a control interception instruction to the proxy subcomponent, so that the proxy subcomponent intercepts and temporarily stores the instructions flowing from the client to the source node; wherein, the data migration component sends a probe key to the source node, so that the source node continues to synchronize the probe key after synchronizing the incremental instructions to the target node, including: the control subcomponent sends the probe key to the source node, so that the source node continues to synchronize the probe key after synchronizing the incremental instructions to the target node; wherein, the data migration... The data migration component detects the probe key at the target node according to the target cycle, including: the control subcomponent detects the probe key at the target node according to the target cycle; wherein, when the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node, including: when the control subcomponent obtains the probe key from the target node, the control subcomponent determines that the source node has completed data synchronization with the target node; wherein, the data migration component sends the intercepted and temporarily stored instructions issued by the client to the target node so as to continue to provide services to the client through the target node, including: the control subcomponent controls the proxy subcomponent to send the intercepted and temporarily stored instructions issued by the client to the target node.

[0010] This disclosure provides a data migration method, comprising: when a source node providing services to a client synchronizes snapshot files to a target node, the client modifies its instruction connection address to a data migration component, so that the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node; and when the source node synchronizes incremental instructions to the target node, it intercepts and temporarily stores the instructions flowing from the client to the source node and determines whether the source node has completed the work of synchronizing data to the target node using a probe key; when it is determined that the source node has completed the work of synchronizing data to the target node, the client modifies its instruction connection address to the target node, so that the client directly sends instructions to the target node; wherein, after it is determined that the source node has completed the work of synchronizing data to the target node, but before the client modifies its instruction connection address to the target node, the data migration component directs the instructions sent by the client to the target node.

[0011] In some embodiments, the client is a backend server providing services to the terminal. The client includes a first server and a second server, and the target node or the source node provides data support to the first server and the second server. The client modifying the instruction connection address to the target node includes: the first server maintaining the instruction connection address connecting to the data migration component unchanged, so that the first server indirectly sends instructions to the target node through the data migration component; and the second server modifying the instruction connection address to the target node, so that the first server directly sends instructions to the target node.

[0012] In some embodiments, the method further includes: after the second server modifies the instruction connection address to the target node, the first server modifies the instruction connection address to the target node, so that the first server directly sends the instruction to the target node.

[0013] This disclosure provides a data migration component, which is configured to: when a source node synchronizes incremental instructions to a target node, intercept and temporarily store instructions sent by a client to the source node, wherein the source node provides services to the client, the incremental instructions are instructions received by the source node from the client after generating a snapshot file in response to a synchronization instruction, and the snapshot file is a snapshot generated by the source node based on existing data in the source node in response to the synchronization instruction, wherein the incremental instructions are synchronized to the target node later than the snapshot file; the data migration component sends a probe key to the source node so that the source node continues to synchronize the probe key after synchronizing the incremental instructions to the target node; the data migration component detects the probe key at the target node according to a target period; when the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node; the data migration component sends the intercepted and temporarily stored instructions sent by the client to the target node so that services can continue to be provided to the client through the target node.

[0014] This disclosure provides a data migration client, which is configured to: when a source node providing services to the client synchronizes snapshot files to a target node, modify the instruction connection address of the client to a data migration component, so that the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node; and when the source node synchronizes incremental instructions to the target node, intercepts and temporarily stores the instructions flowing from the client to the source node and determines whether the source node has completed the work of synchronizing data to the target node through a probe key; when it is determined that the source node has completed the work of synchronizing data to the target node, the client modifies the instruction connection address to the target node, so that the client directly sends instructions to the target node; wherein, after it is determined that the source node has completed the work of synchronizing data to the target node, but before the client modifies the instruction connection address to the target node, the data migration component directs the instructions sent by the client to the target node.

[0015] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing program instructions; and the processor for calling the program instructions stored in the memory to implement the data migration method described above.

[0016] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data migration method as described in any of the preceding embodiments.

[0017] This disclosure provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data migration method described above.

[0018] The data migration method, components, client, electronic device, and computer-readable storage medium provided in this disclosure can intercept and temporarily store the client's instructions pointing to the source node when the source node is synchronizing data to the target node. Then, when it is determined by the probe key that the source node has completed data synchronization with the target node, the data migration component will direct the instructions issued by the client to the target node, thereby completing the data migration from the source node to the target node without the client's awareness. Furthermore, during the data migration process, the simultaneous interception and transfer of instructions by the data migration component ensures that the data obtained by the client is consistent.

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 This is a schematic diagram illustrating the structure of a data migration method based on relevant technologies.

[0022] Figure 2 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0023] Figure 3 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0024] Figure 4 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0025] Figure 5 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0026] Figure 6 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0027] Figure 7 This is a schematic diagram illustrating how a service instruction connection address is modified from the source end to the proxy component according to an exemplary embodiment.

[0028] Figure 8 This is a schematic diagram illustrating a control subcomponent sending an interception instruction to a proxy subcomponent according to an exemplary embodiment.

[0029] Figure 9 This is a schematic diagram illustrating a control subcomponent sending a probe key to a source node and detecting the probe key at a destination node, according to an exemplary embodiment.

[0030] Figure 10 This is a schematic diagram illustrating a control subcomponent sending a switching instruction to a proxy subcomponent and stopping a synchronization tool, according to an exemplary embodiment.

[0031] Figure 11 This is a schematic diagram illustrating how a business instruction connection address is changed from a proxy subcomponent to a target node (such as a Redis node in the cloud) according to an exemplary embodiment.

[0032] Figure 12 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0034] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, elements, steps, etc., can be employed. In other instances, well-known methods, components, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0035] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor components and / or microcontroller components.

[0036] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended inclusion and mean that additional elements / components / etc. may exist besides the listed elements / components / etc.

[0038] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0039] The following section will first explain some of the terms used in the embodiments of this application so that those skilled in the art can understand them.

[0040]

[0041] The preceding text introduced some terms and concepts involved in the embodiments of this application. The following text introduces the technical features involved in the embodiments of this application.

[0042] With the development of cloud computing technology and the continuous increase in the scale of Redis clusters, the deployment form of Redis clusters is also changing. From direct deployment on physical machines in the early days, to deployment based on managed services on cloud computing platforms now, and because the cluster solutions provided by the open-source version of Redis itself are not perfect, cloud vendors will develop their own cluster modes according to the actual situation in the managed services provided. This has created a phenomenon where, although everyone is using Redis, their architectures, versions, etc. are different. In the process of migrating data to the cloud, it is not possible to simply rely on Redis's own master-slave replication mechanism to migrate heterogeneous clusters to the cloud. Therefore, it is necessary to use a cross-cluster data synchronization solution for data migration.

[0043] Commonly used migration and synchronization solutions in the industry include Figure 1 As shown:

[0044] First, the `sync` tool sends a `psync` command (a command requesting synchronization) to the source node (such as the source Redis node). Then, the source node sends an RDB file (a very compact file that stores a snapshot of Redis data at a specific point in time) to the `sync` tool. Next, the source node sends incremental commands to the `sync` tool. Upon receiving the RDB, the `sync` tool parses it into operation commands and sends them to the target node (such as the cloud Redis node). After the RDB is parsed, the tool writes the backlog of incremental commands to the target node (such as the cloud Redis node). Once all the backlog of incremental commands has been written to the target node (such as the cloud Redis node), data synchronization enters real-time incremental synchronization. Clients need to stop writing operations to the source Redis, change the command connection address to the target node (such as the cloud Redis node), and restart all client programs to complete the entire data migration process.

[0045] For data migration in heterogeneous Redis clusters, the main process of existing solutions is as follows:

[0046] 1. Use the sync tool to simulate a slave node (the slave node only acts as a data backup) to synchronize data from the source.

[0047] 2. The sync tool parses the acquired data into commands and writes them to the destination.

[0048] 3. All clients re-enter the network and complete the command connection address switch.

[0049] Existing migration solutions in the industry do not consider the issue of client command switching in their data synchronization tools. When the migration synchronization is complete, the client can only choose one of the following two methods to switch the command from the source end to the target end.

[0050] 1. On the client side, change the Redis access address in the configuration file to the target address, and then restart and go online.

[0051] In this context, the client can refer to the backend server, which can be the client of the underlying service (data).

[0052] 2. Modify the Redis access address in the configuration file of all clients to the target address, shut down the service as a whole, and then restart it.

[0053] Problems with the first switching method:

[0054] Although a smooth switchover will not cause overall business unavailability, the asynchronous switching process of the command connection addresses of various clients may lead to inconsistent migration data, which could have a serious impact.

[0055] For example, during the process of switching the connection address of an instruction, the first client may connect to the source node and the second client may connect to the target node. When requesting to obtain target data (such as the price of a certain product), the first client obtains the data in the source node and the second client obtains the data in the target node.

[0056] Problems with the second switching method:

[0057] Shutting down all clients, modifying the configuration files, and restarting them ensured the consistency of the migrated data. However, shutting down all clients would cause users on the service terminals to be unable to use the service. For some P0-level services, such an interruption is unacceptable.

[0058] This application analyzes the shortcomings of the commonly used migration solutions in the industry and proposes a migration solution that can guarantee both the consistency of the migrated data and the availability of business services, as described below.

[0059] Figure 2 This is a flowchart illustrating a data migration method according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power. For example, the method can be executed by a server or a terminal device, or by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustrative purposes, but this disclosure is not limited thereto.

[0060] Before implementing the following solution, this embodiment will introduce the background of the solution implementation.

[0061] like Figure 1 As shown, before data synchronization in a heterogeneous cluster, the source node receives instructions directly from the client; and upon receiving a synchronization instruction (such as the psync instruction) sent by the synchronization tool, the source node generates a snapshot file (such as an RDB file) based on the data in memory, and continues to receive incremental instructions from the client.

[0062] In this application, when the source node synchronizes snapshot files to the target node through a synchronization tool, the client's command connection address can be modified to the data migration component so that the client sends commands to the data migration component, and then the data migration component transparently forwards the commands sent by the client to the source node.

[0063] It is important to note that when modifying the client's command connection address, a smooth switching method can be used. That is, modify the address of a batch of clients first, and then modify the address of another batch of clients, so that there are always clients providing services to users, thus completing a smooth transition.

[0064] In some embodiments, the client may refer to a backend server for a certain service, and the source or target node may be a cluster that provides data support to the backend server. This smooth transition ensures that there is always a server providing services to the end user, thus preventing any impact on the end user from being affected by changes to the command connection address.

[0065] Reference Figure 2 The data migration method provided in this disclosure may include the following steps.

[0066] In step S202, when the source node synchronizes incremental instructions to the target node, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node. The source node is used to provide services to the client. The incremental instructions are instructions that the source node continues to receive from the client after generating a snapshot file in response to the synchronization instructions. The snapshot file is a snapshot generated by the source node in response to the synchronization instructions based on the data already in the source node. The incremental instructions are synchronized to the target node later than the snapshot file.

[0067] In some embodiments, when the source node starts synchronizing data to the target node, it first synchronizes snapshot files to the target node using a synchronization tool, and then synchronizes incremental commands to the target node using the same tool.

[0068] When the source node that provides data support to the client starts synchronizing incremental commands to the target node through the synchronization tool, the data migration component can intercept and temporarily store the commands sent by the client to the source node, and no longer forward the commands sent by the client to the source node.

[0069] In step S204, the data migration component sends a probe key to the source node so that the source node can continue to synchronize the probe key after synchronizing the incremental instructions to the target node.

[0070] In some embodiments, the source node synchronizes data to the target node serially and sequentially. Therefore, the source node first synchronizes incremental instructions to the target node, and then synchronizes the probe key after the incremental instructions are synchronized.

[0071] Therefore, the aforementioned probe key can be used to determine whether the source node has synchronized all incremental instructions to the target node.

[0072] Step S206: The data migration component detects the probe key at the target node according to the target cycle.

[0073] Step S208: When the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node.

[0074] Once the data migration component obtains the probe key from the target node, it can determine that all incremental commands that arrived at the source node before the probe key was sent (i.e. before the command was intercepted) have been synchronized to the target node. Therefore, the data migration component can direct the commands sent by the client to the target node, so that while the data is being migrated from the source node to the target node, it will not affect the client.

[0075] In step S210, the data migration component sends the intercepted and temporarily stored instructions issued by the client to the target node so that the target node can continue to provide services to the client.

[0076] In some embodiments, when the source node is synchronizing data to the target node, the data migration component can intercept and temporarily store the client's instructions pointing to the source node; then, the data migration component can send a probe key to determine whether the source node has completed the data synchronization to the target node; finally, if the probe key confirms that the source node has completed the data synchronization, the data migration component can direct the client's instructions to the target node. Through the above method, the data migration from the source node to the target node can be completed without the client's awareness, and the simultaneous interception and transfer of instructions by the data migration component during the data migration process ensures that the data obtained by the client is consistent.

[0077] Figure 3 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0078] refer to Figure 3 The above data migration method may include the following steps.

[0079] In step S302, when the source node synchronizes incremental instructions to the target node, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node. The source node is used to provide services to the client. The incremental instructions are instructions that the source node continues to receive from the client after generating a snapshot file in response to the synchronization instructions. The snapshot file is a snapshot generated by the source node in response to the synchronization instructions based on the data already in the source node. The incremental instructions are synchronized to the target node later than the snapshot file.

[0080] In step S304, the data migration component sends a probe key to the source node so that the source node can continue to synchronize the probe key after synchronizing the incremental instructions to the target node.

[0081] Step S306: The data migration component detects the probe key at the target node according to the target cycle.

[0082] Step S308: When the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node.

[0083] In step S310, the data migration component sends a stop signal to the synchronization tool so that the synchronization tool stops synchronizing the data sent by the source node to the target node, so that the source node cannot send the probe key failure information to the target node.

[0084] In some embodiments, the probe key or other keys have a certain time limit. Therefore, the source node will generate and send probe key failure information after a period of time after the probe key synchronization is completed. The probe key failure information is used to delete the probe key.

[0085] In some embodiments, however, after the data migration is complete, the timeliness of the probe key or other keys may be redefined by the target node, at which point the probe key or other keys should no longer be under the control of the source node.

[0086] Therefore, once the data migration component determines that the source node has finished sending data to the target node through the probe key, the data migration component will send a stop signal to the synchronization tool so that the synchronization tool will no longer synchronize data to the target node, thereby isolating the data in the target node from the source node.

[0087] In step S312, the data migration component sends the intercepted and temporarily stored instructions issued by the client to the target node so that the target node can continue to provide services to the client.

[0088] The technical solution provided in this embodiment can immediately disconnect the connection between the source node and the target node after the source node has synchronized the data to the target node, thereby avoiding the influence of the source node on the data in the target node.

[0089] Figure 4 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0090] In some embodiments, the data migration component may include a proxy subcomponent (such as a Proxy component) and a control subcomponent (such as a Manager component). There may be multiple proxy subcomponents and only one control subcomponent.

[0091] The proxy sub-component has the following functions: receiving client commands and providing command forwarding functionality; intercepting and controlling client commands; and switching the command forwarding address.

[0092] The control subcomponent can have the following functions: sending control signals to the Proxy and sync tools; writing probe keys to the source end and obtaining integrity verification of the completed data replication stream at the destination end.

[0093] refer to Figure 4 The above data migration method may include the following steps.

[0094] In step S402, when the source node synchronizes incremental instructions to the target node, the control subcomponent sends a control interception instruction to the proxy subcomponent so that the proxy subcomponent can intercept and temporarily store the instructions flowing from the client to the source node.

[0095] Step S404: The control sub-component sends a probe key to the source node so that the source node can continue to synchronize the probe key after synchronizing the incremental instructions to the target node.

[0096] Step S406: Control the sub-component to detect the probe key at the target node according to the target cycle.

[0097] In step S408, when the control subcomponent obtains the probe key from the target node, the control subcomponent determines that the source node has completed data synchronization with the target node.

[0098] In step S410, the control subcomponent controls the proxy subcomponent to send the intercepted and temporarily stored instructions issued by the client to the target node.

[0099] The technical solution provided in this embodiment can intercept and temporarily store the client's instructions pointing to the source node when the source node is synchronizing data to the target node. Then, after confirming that the source node has completed synchronizing data with the target node through the probe key, the data migration component directs the client's instructions to the target node, thereby completing the data migration from the source node to the target node without the client's awareness. Furthermore, the simultaneous interception and transfer of instructions by the data migration component during the data migration process ensures that the data obtained by the client is consistent.

[0100] Figure 5 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0101] refer to Figure 5 The above data migration method may include the following steps.

[0102] In step S502, when the source node providing services to the client synchronizes snapshot files to the target node, the client modifies the instruction connection address to the data migration component so that the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node. When the source node synchronizes incremental instructions to the target node, the component intercepts and temporarily stores the instructions flowing from the client to the source node and determines whether the source node has completed the work of synchronizing data to the target node through the probe key.

[0103] In step S504, when it is determined that the source node has completed the work of synchronizing data with the target node, the client modifies the command connection address to the target node so that the client can send commands directly to the target node.

[0104] In step S506, after determining that the source node has completed the work of synchronizing data with the target node, before the client modifies the instruction connection address to the target node, the data migration component will direct the instruction sent by the client to the target node.

[0105] The technical solution provided in this embodiment can intercept and temporarily store the client's instructions pointing to the source node when the source node is synchronizing data to the target node. Then, after confirming that the source node has completed synchronizing data with the target node through the probe key, the data migration component directs the client's instructions to the target node, thereby completing the data migration from the source node to the target node without the client's awareness. Furthermore, the simultaneous interception and transfer of instructions by the data migration component during the data migration process ensures that the data obtained by the client is consistent.

[0106] Figure 6 This is a flowchart illustrating a data migration method according to an exemplary embodiment.

[0107] In some embodiments, the client is a backend server that provides services to the terminal. The client includes a first server and a second server, and the target node or the source node provides data support to the first server or the second server.

[0108] refer to Figure 6 The above data migration method may include the following steps.

[0109] In step S602, the first server keeps the instruction connection address of the data migration component unchanged so that the first server can indirectly send instructions to the target node through the data migration component.

[0110] In step S604, the second server modifies the instruction connection address to the target node so that the first server can send the instruction directly to the target node.

[0111] Step S606: After the second server modifies the instruction connection address to the target node, the first server modifies the instruction connection address to the target node so that the first server can send the instruction directly to the target node.

[0112] Using the above method, the modification of the command connection address of each client can be completed smoothly, ensuring that there are always clients providing services to the terminal normally, thus not affecting the user experience.

[0113] The following is an example that can solve the problem of how to achieve a seamless and smooth connection switch between the old and new clusters without service interruption during data migration of heterogeneous Redis cluster nodes. That is, if an application has 10 replicas (such as clients) using the Redis cluster (backend database) at the same time, the user can always have a replica that can provide services normally during the entire data migration process, and the consistency of the cluster data obtained by each replica can be guaranteed.

[0114] In this embodiment, during the data migration process, a Proxy (a proxy sub-component) can be introduced to intercept and control user commands, and a Manager (a control sub-component) can be introduced to perform integrity detection on the Redis replication stream. This enables seamless data migration of heterogeneous clusters without interrupting business operations. The following will demonstrate... Figures 7-11 The overall workflow is shown.

[0115] In this embodiment of the application, by introducing a command relay component Proxyy (a proxy sub-component) to intercept and control user commands and a coordination component Manager (a control sub-component) to perform integrity detection on the Redis replication stream during the data migration process, seamless data migration of heterogeneous clusters can be achieved without interrupting business operations.

[0116] First, let's introduce the functions and roles of the two components: Proxy (a proxy sub-component) and Manager (a control sub-component).

[0117] Proxy component

[0118] 1. Accept client commands and provide command forwarding functionality.

[0119] 2. Intercept and control client commands.

[0120] 3. Switch the address on the instruction forwarding side.

[0121] Manager component

[0122] 1. Send control signals to the Proxy and sync tools.

[0123] 2. Write the probe key to the source end and obtain the integrity verification of the completed data replication stream at the destination end.

[0124] After introducing the two components in the embodiments of this application, the overall workflow of the data migration system will be described below:

[0125] 1. For example Figure 7As shown, at the beginning of the migration, the sync tool is configured to synchronize data from the source Redis (a type of source node) to the destination Redis (a type of target node), while the client command connection address is modified to reach the Proxy component (a type of proxy sub-component).

[0126] 2. At this point, the Proxy component will transparently (without any modification) direct the client's commands to the source Redis node, waiting for the sync tool to reach the incremental command synchronization phase (see reference). Figure 7 ).

[0127] 3. When the sync tool reaches the incremental command synchronization phase, it sends a control signal to the Proxy through the Manager component, causing the Proxy to enter the command interception state. In this state, the Proxy will temporarily store the received client commands but will not forward them to the backend stage nodes (see details). Figure 8 ).

[0128] 4. After the proxy intercepts the commands, it sends a probe key to the source node through the Manager component, while continuously retrieving this probe key from the target node. When the probe key is obtained, it indicates that all write commands to the source Redis before the proxy interception operation have been synchronized to the destination Redis, because Redis replication streams are executed serially and sequentially. (Reference) Figure 9 ).

[0129] 5. When the Manager retrieves the probe key sent to the source on the destination, it indicates that the source Redis cluster has no backlog of write commands. Switching data commands at this point ensures data consistency. To prevent expired key del commands from being synchronized from the source to the destination, the Manager signals the Proxy to switch commands and also signals the sync tool to stop working. This completes the most crucial step in the entire solution—command switching (see reference). Figure 10 ).

[0130] The command switch is complete, and the actual user operation commands have reached the target node. The sync tool has also stopped working. At this point, the client's connection request is still on the proxy. A command connection address switch is still needed for the client to complete the transition from the proxy to the target node's command connection address. This completes the entire data migration process (see reference). Figure 11 ).

[0131] In this embodiment of the application, in the data migration of a heterogeneous Redis cluster, a Proxy component is introduced to take over the client's instructions during the migration process, and a Manager component is introduced to control and coordinate the Proxy and sync tools. This allows the client's instructions to be migrated seamlessly from the source to the target node. At the same time, the Manager performs key detection during the Proxy interception to ensure data consistency, thereby realizing a solution for seamless data migration in a heterogeneous Redis cluster.

[0132] Based on the same inventive concept, this disclosure also provides a data migration component, as shown in the following embodiments. Since the principle by which this component solves the problem is similar to that of the method embodiments described above, the implementation of this component embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0133] This disclosure provides a data migration component that can be used to: intercept and temporarily store instructions sent by a client to a source node when a source node synchronizes incremental instructions to a target node. The source node provides services to the client, and the incremental instructions are instructions received by the source node from the client after generating a snapshot file in response to a synchronization instruction. The snapshot file is a snapshot generated by the source node based on existing data in the source node after responding to the synchronization instruction. The incremental instructions are synchronized to the target node later than the snapshot file. The data migration component sends a probe key to the source node so that the source node continues to synchronize the probe key after synchronizing the incremental instructions to the target node. The data migration component detects the probe key at the target node according to a target period. When the data migration component detects the probe key at the target node, it determines that the source node has completed data synchronization with the target node. The data migration component sends the intercepted and temporarily stored instructions sent by the client to the target node so that services can continue to be provided to the client through the target node.

[0134] In some embodiments, the data migration component can also be used to receive instructions from the client and transparently forward the instructions sent by the client to the source node when the source node synchronizes snapshot files to the target node before the source node synchronizes incremental instructions to the target node; wherein, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node, including: the data migration component intercepts and temporarily stores the instructions sent by the client to the source node; the data migration component does not forward the instructions sent by the client to the source node.

[0135] In some embodiments, the source node synchronizes the probe key to the target node through a synchronization tool. After a period of time following the completion of probe key synchronization, the source node generates and sends probe key failure information, which is used to delete the probe key. Data migration can also be used to send a stop signal to the synchronization tool after the data migration component determines that the source node has completed data synchronization with the target node, so that the synchronization tool will no longer synchronize the data sent by the source node to the target node, and the source node will not send probe key failure information to the target node.

[0136] In some embodiments, the data migration component includes a control subcomponent and a proxy subcomponent; wherein, when the source node synchronizes incremental instructions to the target node, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node, including: when the source node synchronizes incremental instructions to the target node, the control subcomponent sends a control interception instruction to the proxy subcomponent, so that the proxy subcomponent intercepts and temporarily stores the instructions flowing from the client to the source node; wherein, the data migration component sends a probe key to the source node, so that the source node continues to synchronize the probe key after synchronizing incremental instructions to the target node, including: the control subcomponent sends a probe key to the source node, so that the source node continues to synchronize the probe key after synchronizing incremental instructions to the target node; wherein, The data migration component detects probe keys at the target node according to the target cycle, including: the control subcomponent detects probe keys at the target node according to the target cycle; wherein, when the data migration component detects a probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node, including: when the control subcomponent obtains a probe key from the target node, the control subcomponent determines that the source node has completed data synchronization with the target node; wherein, the data migration component sends the intercepted and temporarily stored instructions issued by the client to the target node so that it can continue to provide services to the client through the target node, including: the control subcomponent controls the proxy subcomponent to send the intercepted and temporarily stored instructions issued by the client to the target node.

[0137] This disclosure provides a data migration client, which is used to: when a source node providing services to the client synchronizes snapshot files to a target node, modify the instruction connection address of the client to a data migration component, so that the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node; and when the source node synchronizes incremental instructions to the target node, intercepts and temporarily stores the instructions flowing from the client to the source node and determines whether the source node has completed the work of synchronizing data to the target node through a probe key; when it is determined that the source node has completed the work of synchronizing data to the target node, the client modifies the instruction connection address to the target node, so that the client directly sends instructions to the target node; wherein after it is determined that the source node has completed the work of synchronizing data to the target node, before the client modifies the instruction connection address to the target node, the data migration component will direct the instructions sent by the client to the target node.

[0138] In some embodiments, the client is a backend server that provides services to the terminal. The client includes a first server and a second server, and the target node or the source node provides data support to the first server and the second server. The client modifies the instruction connection address to the target node, which includes: the first server keeping the instruction connection address of the data migration component unchanged so that the first server indirectly sends instructions to the target node through the data migration component; and the second server modifying the instruction connection address to the target node so that the first server directly sends instructions to the target node.

[0139] In some embodiments, the client is further configured to: after the second server modifies the instruction connection address to the target node, the first server modifies the instruction connection address to the target node so that the first server directly sends the instruction to the target node.

[0140] It is understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0142] Figure 12 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 12 The illustrated electronic device 1200 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0143] like Figure 12As shown, the electronic device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage section 1208 into a random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device 1200. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0144] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0145] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs the functions defined above in the system of this application.

[0146] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, component, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, component, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, component, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0147] In another aspect, this application also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: when a source node providing data support to a client begins synchronizing incremental instructions to a target node via a synchronization tool, a data migration component intercepts and temporarily stores the instructions flowing from the client to the source node, wherein the incremental instructions are instructions received by the source node from the client after generating a snapshot file, and the snapshot file is a snapshot generated by the source node in response to the synchronization instructions based on memory data; the data migration component sends a probe key to the source node so that the source node synchronizes the probe key while synchronizing incremental instructions to the target node; the data migration component continuously acquires probe keys at the target node; when the data migration component acquires a probe key from the target node, the data migration component directs the instructions issued by the client to the target node so as to provide services to the client through the target node.

[0148] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0149] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or smart device, etc.) to execute the method according to the embodiments of this disclosure.

[0150] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0151] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A data migration method, characterized in that, include: When the source node synchronizes incremental instructions to the target node, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node, wherein the source node is used to provide services to the client, the incremental instructions are instructions that the source node continues to receive from the client after generating a snapshot file in response to the synchronization instructions, the snapshot file is a snapshot generated by the source node in response to the synchronization instructions based on the data already in the source node, wherein the incremental instructions are synchronized to the target node later than the snapshot file; The data migration component sends a probe key to the source node so that the source node can continue to synchronize the probe key after synchronizing the incremental instruction to the target node; The data migration component detects the probe key at the target node according to the target cycle; When the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node; The data migration component intercepts and temporarily stores the instructions issued by the client and sends them to the target node so that the target node can continue to provide services to the client.

2. The method according to claim 1, characterized in that, Before the source node synchronizes incremental instructions to the target node, the method further includes: When the source node synchronizes the snapshot file to the target node, the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node; The data migration component intercepts and temporarily stores instructions sent by the client to the source node, including: The data migration component intercepts and temporarily stores the instructions sent by the client to the source node; The data migration component does not forward the instructions sent by the client to the source node.

3. The method according to claim 1, characterized in that, The source node synchronizes the probe key to the target node using a synchronization tool. A period of time after the probe key synchronization is complete, the source node generates and sends a probe key failure message, which is used to delete the probe key. The method further includes, after the data migration component determines that the source node has completed data synchronization with the target node: The data migration component sends a stop signal to the synchronization tool so that the synchronization tool stops synchronizing the data sent by the source node to the target node, so that the source node cannot send the probe key failure information to the target node.

4. The method according to claim 1, characterized in that, The data migration component includes a control subcomponent and a proxy subcomponent; wherein, when the source node synchronizes incremental instructions to the target node, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node, including: When the source node synchronizes the incremental instruction to the target node, the control subcomponent sends a control interception instruction to the proxy subcomponent so that the proxy subcomponent can intercept and temporarily store the instruction flowing from the client to the source node; The data migration component sends a probe key to the source node so that the source node can continue synchronizing the probe key after synchronizing the incremental command to the target node, including: The control subcomponent sends the probe key to the source node so that the source node can continue to synchronize the probe key after synchronizing the incremental instruction to the target node; The data migration component detects the probe key at the target node according to the target cycle, including: The control subcomponent detects the probe key at the target node according to the target cycle; Specifically, when the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node, including: When the control subcomponent obtains the probe key from the target node, the control subcomponent determines that the source node has completed data synchronization with the target node; The data migration component intercepts and temporarily stores instructions issued by the client and sends them to the target node so that the target node can continue to provide services to the client, including: The control subcomponent controls the proxy subcomponent to send the intercepted and temporarily stored instructions issued by the client to the target node.

5. A data migration method, characterized in that, include: When the source node providing services to the client synchronizes snapshot files to the target node, the client modifies the instruction connection address to the data migration component, so that the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node, and intercepts and temporarily stores the instructions flowing from the client to the source node when the source node synchronizes incremental instructions to the target node, and determines whether the source node has completed the work of synchronizing data to the target node through a probe key; Once it is determined that the source node has completed the work of synchronizing data with the target node, the client modifies the instruction connection address to the target node so that the client can send instructions directly to the target node; After determining that the source node has completed the data synchronization process with the target node, before the client modifies the instruction connection address to the target node, the data migration component will direct the instruction sent by the client to the target node.

6. The method according to claim 5, characterized in that, The client is a backend server providing services to the terminal. The client includes a first server and a second server. The target node or the source node provides data support to the first server and the second server. The client modifies the instruction connection address to the target node, including: The first server keeps the instruction connection address of the data migration component unchanged, so that the first server can indirectly send instructions to the target node through the data migration component; The second server modifies the instruction connection address to the target node so that the first server can send the instruction directly to the target node.

7. The method according to claim 6, characterized in that, The method further includes: After the second server modifies the instruction connection address to the target node, the first server modifies the instruction connection address to the target node so that the first server can send the instruction directly to the target node.

8. A data migration component, characterized in that, The data migration component is used for: When the source node synchronizes incremental instructions to the target node, the data migration component intercepts and temporarily stores the instructions sent by the client to the source node, wherein the source node is used to provide services to the client, the incremental instructions are instructions that the source node continues to receive from the client after generating a snapshot file in response to the synchronization instructions, the snapshot file is a snapshot generated by the source node in response to the synchronization instructions based on the data already in the source node, wherein the incremental instructions are synchronized to the target node later than the snapshot file; The data migration component sends a probe key to the source node so that the source node can continue to synchronize the probe key after synchronizing the incremental instruction to the target node; The data migration component detects the probe key at the target node according to the target cycle; When the data migration component detects the probe key at the target node, the data migration component determines that the source node has completed data synchronization with the target node; The data migration component intercepts and temporarily stores the instructions issued by the client and sends them to the target node so that the target node can continue to provide services to the client.

9. A data migration client, characterized in that, The data migration client is used for: When the source node providing services to the client synchronizes snapshot files to the target node, the client modifies the instruction connection address to the data migration component, so that the data migration component receives instructions from the client and transparently forwards the instructions sent by the client to the source node, and intercepts and temporarily stores the instructions flowing from the client to the source node when the source node synchronizes incremental instructions to the target node, and determines whether the source node has completed the work of synchronizing data to the target node through a probe key; Once it is determined that the source node has completed the work of synchronizing data with the target node, the client modifies the instruction connection address to the target node so that the client can send instructions directly to the target node; After determining that the source node has completed the data synchronization process with the target node, before the client modifies the instruction connection address to the target node, the data migration component will direct the instruction sent by the client to the target node.

10. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being used to execute the data migration method as described in any one of claims 1-7 based on instructions stored in the memory.

11. A computer-readable storage medium having stored thereon program instructions that, when executed by a processor, implement the data migration method as described in any one of claims 1-7.

Images