Data migration method and system

Abstract

The invention relates to a data migration method and system, and belongs to the technical field of data processing.The method comprises the steps that a first Master node is deployed in a first machine room, and under the condition that the first Master node is initialized, cluster information and metadata information of a second Master node are synchronized to the first Master node based on connection between the second Master node in a second machine room and the first Master node; under the condition of the first limited routing rule, deploying a coordination node, a data node and a first consumption cluster in the first machine room; and migrating the data of the data node in the second machine room to the data node in the first machine room under the condition of carrying out rollover operation on the index. According to the method, the migration of the large-scale cluster is realized, the migration cost is reduced, the complexity of technical implementation is reduced, and the influence on the outside is minimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data migration method and system. Background Art

[0002] Among related technologies, Elasticsearch is currently the most popular database for real-time, massive log collection. Developers can use Elasticsearch to perform real-time queries and compare historical data, allowing them to promptly identify system issues. Logging systems based on Elasticsearch can be horizontally scaled to incorporate more nodes and support more data. However, for such a large system, if, for unforeseen reasons, it needs to be migrated to another data center, it is even more challenging to ensure that the migration remains completely transparent to the outside world.

[0003] Currently, the official method given by elasticsearch is to use CCR (Cross-cluster replication). This feature is a paid item and requires the purchase of a platinum membership. The free method is more complicated and requires higher requirements for operation and maintenance personnel. The specific steps are as follows: migrate the index system indexes .security, .kibana through the reindex interface, import the life cycle, index template, etc. to the new cluster, create corresponding indexes and aliases, and then collect real-time data to the new and old clusters at the same time, and finally import the old data remotely through the reindex interface. This method has many steps and is prone to errors. It also refreshes the life cycle of the index, and requires additional scripts to delete expired indexes. Therefore, how to better implement data migration has become an urgent problem to be solved. Summary of the Invention

[0004] In view of the above shortcomings of the existing technology, the purpose of the present invention is to provide a data migration method and system. The method realizes the migration of large-scale clusters, reduces the migration cost, reduces the complexity of technical implementation, and minimizes the impact on the outside world.

[0005] The first aspect of the present invention proposes a data migration method, including: S1, deploying a first Master node in a first computer room, and when the first Master node is initialized, synchronizing the cluster information and metadata information of the second Master node to the first Master node based on the connection between the second Master node and the first Master node in the second computer room; S2, deploying a coordination node, a data node and a first consumer cluster in the first computer room under a first restrictive routing rule; S3, migrating the data of the data node in the second computer room to the data node in the first computer room when performing a rollover operation on the index.

[0006] Furthermore, when the cluster information and metadata information of the second Master node are synchronized to the first Master node, the second Master nodes are shut down one by one.

[0007] Furthermore, the first restricted routing rule includes: storing the cluster index in the second consumer cluster in the second computer room.

[0008] Furthermore, when the coordination node and the data node are deployed in the first computer room, the coordination node and the data node are initialized. When the coordination node and the data node are initialized, the coordination node and the data node are both connected to the first Master node, wherein the coordination node is connected to the first consumer cluster.

[0009] Furthermore, a rollover operation is performed on the index, including: based on the rollover operation on the index, modifying the routing rules of the index template, obtaining the target index and applying the target routing rules; wherein, when the data of the data node in the second computer room is migrated to the data node in the first computer room, the second consumer cluster is closed and the first consumer cluster is enabled.

[0010] Furthermore, migrating the data of the data nodes in the second computer room to the data nodes in the first computer room includes: modifying the routing rules of the index to migrate the data of the data nodes in the second computer room to the data nodes in the first computer room.

[0011] Furthermore, it also includes: when the data of the data node in the second computer room is migrated to the data node in the first computer room, the data node in the second computer room is taken offline; the external domain name of the coordination node in the second computer room is switched to the first computer room, and when the switching is completed, the coordination node in the second computer room is taken offline.

[0012] The second aspect of the present invention proposes a data migration system, including: a synchronization module, which is used to deploy a first Master node in a first computer room, and when the first Master node is initialized, based on the connection between the second Master node and the first Master node in the second computer room, synchronize the cluster information and metadata information of the second Master node to the first Master node; a deployment module, which is used to deploy a coordination node, a data node and a first consumer cluster in the first computer room under a first restrictive routing rule; a migration module, which is used to migrate the data of the data node in the second computer room to the data node in the first computer room when performing a rollover operation on the index.

[0013] In a third aspect of the present invention, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in the first aspect of the present invention.

[0014] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to the first aspect of the present invention.

[0015] The beneficial effects of the present invention are as follows:

[0016] The data migration method and system described in the present invention deploys a first Master node in a first computer room. When the first Master node is initialized, the cluster information and metadata information of the second Master node are synchronized with the first Master node based on the connection between the second Master node and the first Master node in the second computer room. Under the first restricted routing rule, a coordination node, data nodes, and a first consumer cluster are deployed in the first computer room. When a rollover operation is performed on the index, the data of the data node in the second computer room is migrated to the data node in the first computer room. This method realizes the migration of large-scale clusters, reduces migration costs, reduces the complexity of technical implementation, and minimizes the impact on the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals represent the same components. Obviously, the drawings described below are only some of the embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings.

[0018] Figure 1 is a flow chart of a data migration method according to one embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a data migration system according to an embodiment of the present invention;

[0020] Figure 3 It is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present invention.

[0022] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0023] In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with certain aspects of the present invention, as detailed in the appended claims.

[0025] The present invention provides a data migration method, system, and related devices. Specifically, the data migration method, system, and related devices according to embodiments of the present invention are described below with reference to the accompanying drawings.

[0026] Figure 1This is a flow chart of a data migration method according to one embodiment of the present invention. It should be noted that the data migration method according to one embodiment of the present invention can be applied to a data migration system according to one embodiment of the present invention. The data migration system can be configured on an electronic device or on a server. This is not limited to this embodiment of the present application.

[0027] like Figure 1 As shown, the data migration method includes:

[0028] S110, deploy the first Master node in the first computer room. When the first Master node is initialized, based on the connection between the second Master node in the second computer room and the first Master node, synchronize the cluster information and metadata information of the second Master node to the first Master node.

[0029] In an embodiment of the present invention, when the cluster information and metadata information of the second Master node are synchronized to the first Master node, the second Master nodes are shut down one by one.

[0030] Among them, the first computer room can be understood as the new computer room, and the second computer room can be understood as the original computer room.

[0031] S120 , deploying a coordination node, a data node, and a first consumer cluster in a first computer room under a first restricted routing rule.

[0032] Among them, in an embodiment of the present invention, the first restricted routing rule includes: storing the cluster index in the second consumer cluster in the second computer room. That is to say, all the indexes of the cluster can only be routed to the data nodes in the consumer-level group of the original computer room, and the cluster is logically split into two without interfering with each other, so as to avoid migrating the shards immediately after the new node joins, causing bandwidth pressure. That is, to modify the routing rule of the index is to tell the system that the index needs to be stored in a specified set of machines (here specifically refers to the machines fixed to the consumer-level group of the original computer room). If no modification is made, the original index will be evenly distributed to all nodes.

[0033] In an embodiment of the present invention, when the coordination node and the data node are deployed in the first computer room, the coordination node and the data node are initialized. When the coordination node and the data node are initialized, the coordination node and the data node are both connected to the first Master node, wherein the coordination node is connected to the first consumer cluster.

[0034] That is to say, the coordination node and data node of the first computer room point to the first Master node during initialization. At this time, no data will be routed to the new node (i.e., the coordination node and data node of the first computer room). The first consumer cluster uses the same group.id as the second consumer cluster.

[0035] S130 , when performing a rollover operation on the index, migrate the data of the data nodes in the second computer room to the data nodes in the first computer room.

[0036] In an embodiment of the present invention, the implementation method of the rollover operation on the index is: based on the rollover operation on the index, the routing rules of the index template are modified to obtain the target index and apply the target routing rules; wherein, when the data of the data node in the second computer room is migrated to the data node in the first computer room, the second consumer cluster is closed and the first consumer cluster is enabled. That is, the routing rules of the index template are modified and a rollover is required before they can be applied to the new index. Among them, Rollover is an operation on the index, which is used to generate a new index, sharing a rolling alias with the original index. Operations on the rolling alias outside the cluster will remain unchanged. It can be understood as an nginx that provides an IP port to the outside world, but the backend can correspond to multiple servers.

[0037] In an embodiment of the present invention, the routing rules of the index are modified to migrate the data of the data nodes in the second computer room to the data nodes in the first computer room. That is, the modification of the routing rules of the index takes effect immediately on the index.

[0038] In an embodiment of the present invention, when the data of the data node in the second computer room is migrated to the data node in the first computer room, the data node in the second computer room is taken offline; the external domain name of the coordination node in the second computer room is switched to the first computer room, and when the switching is completed, the coordination node in the second computer room is taken offline.

[0039] According to the data migration method of an embodiment of the present invention, by deploying and initializing the first master node in the first computer room and synchronizing the cluster information and metadata of the second master node in the second computer room with the first master node, initial data preparation and synchronization are achieved, laying the foundation for subsequent migration work. After synchronization is complete, the second master nodes are shut down one by one, ensuring system stability and continuity during the migration process and avoiding service unavailability caused by sudden interruptions. By implementing the first restrictive routing rule, the cluster index is stored in the second consumer cluster in the second computer room, avoiding the need to migrate shards immediately after the new node joins, thereby reducing bandwidth pressure and improving migration efficiency. This strategy also logically splits the cluster into two non-interfering parts, ensuring data consistency and service stability during the migration process. By performing a rollover operation on the index, modifying the routing rules of the index template, and applying the target routing rules, flexible data migration is achieved. This operation method is not only simple and easy to implement, but also ensures data integrity and accuracy during the migration process. After the data migration is complete, the second consumer cluster can be conveniently shut down and the first consumer cluster can be enabled, achieving a smooth switch between consumer clusters. During the migration process, by gradually shutting down and taking offline the data nodes and coordination nodes in the old computer room, the resources of the new computer room can be efficiently utilized while avoiding the waste of resources in the old computer room. This migration method also enables the system to adapt more flexibly to the needs of business development, thereby reducing long-term operation and maintenance costs.

[0040] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0041] According to one aspect of an embodiment of the present invention, a data migration system is also proposed. Figure 2 Schematic diagram of a data migration system according to an embodiment of the present invention; Figure 2 As shown, including:

[0042] Synchronization module 210, configured to deploy a first master node in the first computer room and, when the first master node is initialized, synchronize the cluster information and metadata of a second master node in the second computer room with the first master node based on the connection between the second master node and the first master node.

[0043] A deployment module 220 is configured to deploy a coordination node, a data node, and a first consumer cluster in the first computer room under a first restricted routing rule;

[0044] The migration module 230 is used to migrate the data of the data nodes in the second computer room to the data nodes in the first computer room when performing a rollover operation on the index.

[0045] According to the data migration system of an embodiment of the present invention, a first Master node is deployed in a first computer room. When the first Master node is initialized, the cluster information and metadata information of the second Master node are synchronized to the first Master node based on the connection between the second Master node and the first Master node in the second computer room. Under the first restricted routing rule, a coordination node, data nodes, and a first consumer cluster are deployed in the first computer room. When a rollover operation is performed on the index, the data of the data node in the second computer room is migrated to the data node in the first computer room. This method realizes the migration of large-scale clusters, reduces migration costs, reduces the complexity of technical implementation, and minimizes the impact on the outside world.

[0046] Optionally, when the cluster information and metadata information of the second Master node are synchronized to the first Master node, the second Master nodes are shut down one by one.

[0047] Optionally, the deployment module 220 is specifically configured to store the cluster index in the second consumer cluster in the second computer room.

[0048] Optionally, when the coordination node and the data node are deployed in the first computer room, the coordination node and the data node are initialized. When the coordination node and the data node are initialized, the coordination node and the data node are both connected to the first Master node, wherein the coordination node is connected to the first consumer cluster.

[0049] Optionally, the migration module 230 is specifically used to modify the routing rules of the index template based on a rollover operation on the index, obtain the target index and apply the target routing rules; wherein, when the data of the data node in the second computer room is migrated to the data node in the first computer room, the second consumer cluster is closed and the first consumer cluster is enabled.

[0050] Optionally, the migration module 230 is specifically configured to modify the routing rule of the index and migrate the data of the data nodes in the second computer room to the data nodes in the first computer room.

[0051] Optionally, it also includes an offline module, which is used to offline the data nodes in the second computer room when the data migration of the data nodes in the second computer room to the data nodes in the first computer room is completed; switch the external domain name of the coordination node in the second computer room to the first computer room, and offline the coordination node in the second computer room when the switching is completed.

[0052] According to one aspect of an embodiment of the present invention, an electronic device is provided.

[0053] Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 3 As shown, the electronic device may include one or more ( Figure 3 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the electronic device may further include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above terminal device. Figure 3 More or fewer components than shown, or with Figure 3 Equivalent functions or comparisons shown Figure 3 Shown are different configurations with more functionality.

[0054] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data migration method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a communications provider of a switching device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0056] The present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute a data migration method.

[0057] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation plans of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, and is not a limitation on the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

[0058] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0059] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0060] Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that: include: S1, deploying a first master node in the first computer room, and synchronizing cluster information and metadata information of the second master node to the first master node based on the connection between the second master node in the second computer room and the first master node when the first master node is initialized; S2, deploying a coordination node, a data node and a first consumer cluster in the first computer room under the first restriction routing rule; S3: When performing a rollover operation on the index, migrate the data of the data nodes in the second computer room to the data nodes in the first computer room.

2. The data migration method according to claim 1, characterized in that: After the cluster information and metadata information of the second Master node are synchronized to the first Master node, shut down the second Master nodes one by one.

3. The data migration method according to claim 1, characterized in that: The first restricted routing rule includes: The cluster index is stored in the second consumer cluster in the second computer room.

4. The data migration method according to claim 3, characterized in that: When the coordination node and the data node are deployed in the first computer room, the coordination node and the data node are initialized. When the coordination node and the data node are initialized, the coordination node and the data node are both connected to the first Master node, wherein the coordination node is connected to the first consumer cluster.

5. The data migration method according to claim 4, characterized in that: Perform a rollover operation on the index, including: Based on the rollover operation on the index, the routing rules of the index template are modified to obtain the target index and apply the target routing rules; wherein, When the data of the data nodes in the second computer room are migrated to the data nodes in the first computer room, the second consumer cluster is closed and the first consumer cluster is enabled.

6. The data migration method according to claim 5, characterized in that: Migrating data of the data node in the second computer room to the data node in the first computer room includes: Modify the routing rule of the index and migrate the data of the data nodes in the second computer room to the data nodes in the first computer room.

7. The data migration method according to claim 1, characterized in that: Also includes: When the data of the data nodes in the second computer room are migrated to the data nodes in the first computer room, the data nodes in the second computer room are offline; Switch the external domain name of the coordination node in the second computer room to the first computer room, and when the switching is completed, take the coordination node in the second computer room offline.

8. A data migration system, characterized in that: include: A synchronization module, configured to deploy a first master node in the first computer room, and synchronize cluster information and metadata information of the second master node to the first master node based on the connection between the second master node in the second computer room and the first master node when the first master node is initialized; A deployment module, used to deploy a coordination node, a data node and a first consumer cluster in the first computer room under the first restriction routing rule; The migration module is used to migrate the data of the data nodes in the second computer room to the data nodes in the first computer room when a rollover operation is performed on the index.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images