Method and apparatus for inter-cluster data transfer

Abstract

The application discloses a kind of inter-cluster data transfer method and device, it is related to computer technical field.The specific embodiment of the method includes: the association of source cluster and target cluster is established by monitoring mechanism;Source cluster sends the data information to be transferred to monitoring mechanism;Target cluster obtains the data information to be transferred by monitoring mechanism, and creates target table for saving data to be transferred, sends target table information to monitoring mechanism;Source cluster obtains target table information by monitoring mechanism, then sends data to be transferred to target table.The embodiment solves the problem of tedious operation, easy to make mistake etc. caused by manual operation, greatly improves the processing efficiency of operation personnel.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for transferring data between clusters. Background Art

[0002] In practical applications of HBase (a distributed database based on multiple servers), multiple HBase clusters (one cluster typically consists of multiple servers) are typically deployed. Sometimes, to reduce the access pressure on a cluster, it is necessary to migrate some of the services on that cluster to other clusters with less traffic. During this process, ensuring data integrity (i.e., preventing data loss) and stability is crucial. This process is called inter-cluster data flow or data transfer.

[0003] Currently, data transfer between clusters is primarily performed manually. This involves packaging the source cluster's data and generating a snapshot. This snapshot is then manually sent to the target cluster via commands, enabling external services. Finally, the relevant data in the source cluster is manually deleted via commands to achieve data transfer between clusters.

[0004] In the process of implementing the present invention, the inventors discovered that the prior art has at least the following problems:

[0005] During manual operation, the process is relatively cumbersome and may fail due to operational errors. Summary of the Invention

[0006] In view of this, the embodiments of the present invention provide a method and apparatus for inter-cluster data migration, which can solve the problems of cumbersome and error-prone operations caused by manual operations, and greatly improve the processing efficiency of operation and maintenance personnel.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for transferring data between clusters is provided.

[0008] A method for transferring data between clusters comprises: establishing an association between a source cluster and a target cluster through a monitoring mechanism; the source cluster sends data information to be transferred to the monitoring mechanism; the target cluster obtains the data information to be transferred through the monitoring mechanism, creates a target table for storing the data to be transferred, and sends the target table information to the monitoring mechanism; the source cluster obtains the target table information through the monitoring mechanism, and then sends the data to be transferred to the target table.

[0009] Optionally, establishing the association between the source cluster and the target cluster through the monitoring mechanism includes: creating a first transient node on the monitoring mechanism to identify the source cluster and the target cluster to establish the association.

[0010] Optionally, the source cluster sending the data information to be transferred to the monitoring mechanism includes: the source cluster sending the data information to be transferred to the monitoring mechanism by using a command parser command or calling an external interface.

[0011] Optionally, after the source cluster sends the data information to be transferred to the monitoring organization, it also includes: the monitoring organization creates a second transient node for storing the data information to be transferred, and the data information to be transferred includes a target cluster identifier; and the target cluster obtains the data information to be transferred through the monitoring organization, including: the target cluster obtains the data information to be transferred by listening to the second transient node related to the target cluster on the monitoring organization.

[0012] Optionally, the source cluster sending the data to be transferred to the target table includes: the source cluster asynchronously sending the data to be transferred to the target table through a remote procedure call according to a mutual trust agreement between the source cluster and the target cluster.

[0013] Optionally, the monitoring mechanism monitors the status of the source cluster and the target cluster, as well as the data location transferred by the source cluster and the data location received by the target cluster. Moreover, during the data transfer process, if a node status abnormality occurs in the cluster, the monitoring mechanism notifies the cluster where the node status abnormality occurs to use other nodes for retry.

[0014] Optionally, the method also includes: when the data transfer is completed, the source cluster deletes the data to be transferred and reports deletion completion information to the monitoring organization; the target cluster obtains the deletion completion information reported by the source cluster through the monitoring organization, and provides services for the data to be transferred; and disconnects the association between the source cluster and the target cluster through the monitoring organization.

[0015] According to another aspect of an embodiment of the present invention, a device for transferring data between clusters is provided.

[0016] A device for transferring data between clusters comprises: an association establishing module for establishing an association between a source cluster and a target cluster through a monitoring mechanism; an information sending module for sending data information to be transferred from the source cluster to the monitoring mechanism; an information processing module for obtaining the data information to be transferred from the target cluster through the monitoring mechanism, creating a target table for storing the data to be transferred, and sending the target table information to the monitoring mechanism; and a data transfer module for obtaining the target table information from the source cluster through the monitoring mechanism and then sending the data to be transferred to the target table.

[0017] According to yet another aspect of an embodiment of the present invention, an electronic device for transferring data between clusters is provided.

[0018] An electronic device for inter-cluster data transfer includes: one or more processors; and a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the inter-cluster data transfer method provided in an embodiment of the present invention.

[0019] According to yet another aspect of an embodiment of the present invention, a computer-readable medium is provided.

[0020] A computer-readable medium stores a computer program, which, when executed by a processor, implements a method for inter-cluster data transfer provided by an embodiment of the present invention.

[0021] One embodiment of the above invention has the following advantages or beneficial effects: establishing an association between a source cluster and a target cluster through a monitoring mechanism; the source cluster sends the data information to be transferred to the monitoring mechanism; the target cluster obtains the data information to be transferred through the monitoring mechanism, creates a target table for storing the data to be transferred, and sends the target table information to the monitoring mechanism; the source cluster obtains the target table information through the monitoring mechanism, and then sends the data to be transferred to the target table, thereby realizing intelligent transfer of data between clusters based on the monitoring mechanism without the need for manual intervention, solving the problems of cumbersome and error-prone operations caused by manual operations, and greatly improving the processing efficiency of operation and maintenance personnel. At the same time, the monitoring mechanism can also monitor the cluster status and the status during the data transfer process, so as to facilitate timely detection of problems and retry, thereby ensuring that data transfer can be carried out safely and effectively.

[0022] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0024] Figure 1 1 is a schematic diagram of the main steps of a method for inter-cluster data transfer according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of the system architecture of an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of main modules of an apparatus for inter-cluster data transfer according to an embodiment of the present invention;

[0027] Figure 4 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0028] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] Figure 1 FIG. 1 is a schematic diagram of the main steps of the method for transferring data between clusters according to an embodiment of the present invention. Figure 1 As shown, the method for inter-cluster data migration according to the embodiment of the present invention mainly includes the following steps S101 to S104.

[0031] Step S101: establishing an association between a source cluster and a target cluster through a monitoring mechanism;

[0032] Step S102: The source cluster sends the data information to be transferred to the monitoring organization;

[0033] Step S103: The target cluster obtains the data information to be transferred through the monitoring organization, creates a target table for storing the data to be transferred, and sends the target table information to the monitoring organization;

[0034] Step S104: the source cluster obtains target table information through a monitoring mechanism, and then sends the data to be transferred to the target table.

[0035] Because the source and target clusters are independent (i.e., unconnected), data transfer in this state requires establishing a connection between the two clusters through an intermediary (i.e., the monitoring mechanism in this embodiment of the present invention). Establishing a connection between the clusters has the following main functions:

[0036] 1. The source cluster and the target cluster establish a mutual trust relationship to facilitate authorized communication when data transfer occurs;

[0037] 2. When data transfer occurs, the monitoring mechanism can be used to monitor the status of the data transfer process, such as the amount of data that has been transferred and the amount of data that still needs to be transferred, the location of the data sent by the source cluster, the location of the data received by the target cluster, and whether the status of the two clusters is normal, etc.

[0038] According to one embodiment of the present invention, when establishing an association between the source cluster and the target cluster through a monitoring mechanism, step S101 may specifically be to identify the source cluster and the target cluster by creating a first transient node on the monitoring mechanism to establish an association. The transient node is a temporary node used to perform monitoring tasks. By monitoring the transient node, the source cluster and the target cluster can obtain the node change content in a timely manner and perform corresponding processing when the transient node changes. The first transient node stores the data of the source cluster and the target cluster, such as: the status of the cluster, the cluster address, and so on. Generally, since a cluster usually includes multiple nodes, the status of the cluster refers to the status of the multiple nodes included in the cluster, the cluster address refers to the addresses of the multiple nodes included in the cluster, and so on.

[0039] After the source cluster and the target cluster are associated through the monitoring organization, they will regularly report cluster status information to the monitoring organization so that the monitoring organization can promptly detect status anomalies and make scheduling adjustments to ensure that data transfer can be carried out safely and smoothly.

[0040] After the source cluster and the target cluster are associated, data transfer between the two clusters can be performed. When performing data transfer, the source cluster first initiates the data transfer. According to another embodiment of the present invention, when the source cluster sends the data information to be transferred to the monitoring agency, it can specifically send the data information to be transferred to the monitoring agency through a command parser command or by calling an external interface. Among them, the command parser command is a shell command, and the data information to be transferred includes, for example: a data table identifier that records the data to be transferred, a target cluster identifier, and so on. For example: the shell command ". / runtablename target cluster address", which includes the data table identifier tablename of the data to be transferred, and the target cluster identifier (target cluster address). Among them, each time data is transferred, it may be one table or multiple tables, so the data table identifier can be one or multiple.

[0041] According to another embodiment of the present invention, after the source cluster sends data information to be transferred to the monitoring organization, the monitoring organization creates a second transient node to store the data information to be transferred. The data information to be transferred includes the identifier of the target cluster. After receiving the data information to be transferred, the monitoring organization creates a second transient node to record information about the data transfer process, such as the amount of data to be sent, the location of data sent by the source cluster, and the location of data received by the target cluster.

[0042] Furthermore, when the target cluster obtains the data information to be transferred through the monitoring mechanism, the target cluster obtains the data information to be transferred by monitoring the second transient node related to the target cluster on the monitoring mechanism. After the monitoring mechanism creates the second transient node, since the target cluster identifier is stored in the second transient node, the target cluster will monitor the creation of the second transient node and obtain the data information to be transferred. Then, the target cluster will create a data table (target table) to store the data to be transferred. Furthermore, after the target table is created, the target table information (such as the target table identifier, target table address, etc.) will be reported to the monitoring mechanism so that the monitoring mechanism can modify the data stored in the second transient node.

[0043] Afterwards, the source cluster monitors that the data of the second instantaneous node has changed, and starts to transfer data, and the source cluster sends the data to be transferred to the target table. In another embodiment of the present invention, the source cluster sends the data to be transferred to the target table asynchronously in accordance with the mutual trust agreement between the source cluster and the target cluster through remote procedure calls. Specifically, the source cluster will open a data generation thread pool, and the target cluster will open a data receiving thread pool. The source cluster needs to first select a node of the target cluster as the receiving data source. It can first obtain all or part of the nodes of the target cluster through a monitoring mechanism, and then select a node from them; then, through a remote procedure call RPC (Remote Procedure Call), the node of the source cluster sends the data to the selected node of the target cluster. In the process of data transfer, data transfer is carried out in accordance with the mutual trust agreement between the source cluster and the target cluster, wherein the mutual trust agreement between clusters is, for example, the secure shell protocol ssh (Secure Shell) or the secure transmission protocol possh.

[0044] After the source cluster sends data to the target cluster, it reports the location of the data to the monitoring organization. After receiving the data from the source cluster, the target cluster saves the data to the target table and reports the location of the received data to the monitoring organization so that the monitoring organization can update the data of the second instantaneous node.

[0045] The monitoring mechanism monitors the status of the source and target clusters through the first transient node and the location of data sent by the source cluster and received by the target cluster through the second transient node. If a node in the cluster experiences an anomaly during the data transfer process, the monitoring mechanism notifies the cluster with the anomaly to retry using another node. Both the source and target clusters may experience anomalies. In this case, the monitoring mechanism will detect the cluster node anomaly and automatically transfer the ongoing data transfer task to another node for retry to continue processing.

[0046] According to another embodiment of the present invention, after the source cluster sends the last piece of data, it reports the completion of the transmission to the monitoring mechanism, for example, by reporting a "complete" message. After the target cluster receives the last piece of data from the source cluster, it reports the completion of the reception to the monitoring mechanism, for example, by reporting a "complete" message, to indicate the completion of the data transfer. The monitoring mechanism updates the data of the second transient node based on the data reported by the source and target clusters.

[0047] When the data transfer is complete (i.e., when the source cluster receives the target cluster's notification of completion), it deletes the data to be transferred and reports the deletion completion information to the monitoring organization. The target cluster receives the deletion completion information from the source cluster through the monitoring organization and provides services for the data to be transferred. The monitoring organization then disconnects the source and target clusters. The monitoring organization terminates the data transfer by deleting the second transient node and disconnects the source and target clusters by deleting the first transient node.

[0048] Figure 2 Schematic diagram of the system architecture of an embodiment of the present invention. Figure 2 As shown, in an embodiment of the present invention, a monitoring mechanism is used to establish an association between the source cluster and the target cluster, and the status of the source cluster and the target cluster is monitored so that when any cluster has an abnormal status, it can be discovered in time to ensure the safety and smooth progress of data transfer. In an embodiment of the present invention, the monitoring mechanism is, for example, the distributed application coordination service Zookeeper (hereinafter referred to as zk). Combined with Figure 2 The system architecture shown in FIG, the data transfer process of the embodiment of the present invention is as follows:

[0049] 1. Create a first instantaneous node on the monitoring organization zk to identify the source cluster and establish an association with the target cluster;

[0050] 2. Save the data of the two clusters in the first instantaneous node, including the cluster monitoring status, cluster address, etc.

[0051] 3. The source cluster and target cluster report monitoring status information to ZK regularly;

[0052] 4. Send data transfer instructions to the monitoring organization zk through the source cluster. On the source cluster side, data transfer instructions can be sent through shell commands or by calling external interfaces. The data transfer instructions include: the data table information included in the data transfer and the target cluster identifier, etc., the data to be transferred;

[0053] 5. After receiving the data to be transferred reported by the source cluster, the monitoring mechanism zk will create a second transient node to monitor the data during the data transfer process, including the location of the data sent by the source cluster, the location of the data received by the target cluster, the amount of data to be sent, etc.

[0054] 6. After the monitoring organization completes the creation of the second transient node, the target cluster monitors the creation of the second transient node. At this time, the target cluster will obtain the data information to be transferred from the second transient node and create a corresponding target table to store the received data. Then, the target cluster reports the created target table information to the monitoring organization zk so that zk can update the data of the second transient node.

[0055] 7. The source cluster and the target cluster perform data transfer. The source cluster monitors the data of the second instant node on ZK and starts to transfer data. The main process is as follows:

[0056] (1) During the data transfer process, it is completed through asynchronous threads. The source cluster will start a data sending thread pool, and the target cluster will start a data receiving thread pool;

[0057] (2) After the data transfer task is started, the source cluster will actively search for the node in the target cluster to receive data. For the source cluster, it is necessary to first select a server node in the target cluster as the receiving data source, and then use RPC (the underlying layer is implemented through socket streams and combined with Google's protobuf open source, the two are implemented in combination) to send the data from the server node of the source cluster to the server node of the target cluster. At the same time, the location of the sent data is reported to ZK. In the process of data transfer, it should be carried out in accordance with the mutual trust agreement between the source cluster and the target cluster;

[0058] (3) After receiving the data, the target cluster saves the data and reports the received data location to ZK;

[0059] (4) During the data transfer process, if a service node in any cluster experiences an abnormality, the monitoring organization will notify the abnormal cluster to replace other service nodes and retry;

[0060] 8. Data transfer completion processing. When data transfer is complete, the source cluster, target cluster, and monitoring organization perform the following processing:

[0061] (1) After the source cluster sends the last piece of data, it needs to report a "complete" data to the monitoring organization zk so that the monitoring organization zk can modify the data of the second instantaneous node;

[0062] (2) After the target cluster receives the last piece of data sent by the source cluster, it reports a "complete" piece of data to the monitoring organization zk, so that the monitoring organization zk can modify the data of the second instantaneous node;

[0063] (3) After the source cluster monitors the data change of the second instantaneous node, it deletes the data that has been transferred to the cluster and deletes the relevant data tables. It then reports the deletion completion to the monitoring organization zk so that the monitoring organization zk can modify the data of the second instantaneous node;

[0064] (4) After the target cluster monitors the data change of the second instantaneous node, it opens the external service for the transferred data and accepts external access;

[0065] (5) The monitoring organization zk deletes the first transient node to end the data transfer and deletes the second transient node to disconnect the source cluster from the target cluster.

[0066] Figure 3 FIG. 1 is a schematic diagram of the main modules of the device for inter-cluster data transfer according to an embodiment of the present invention. Figure 3 As shown, the apparatus 300 for inter-cluster data transfer according to the embodiment of the present invention mainly includes an association establishing module 301 , an information sending module 302 , an information processing module 303 and a data transfer module 304 .

[0067] An association establishing module 301 is used to establish an association between a source cluster and a target cluster through a monitoring mechanism;

[0068] An information sending module 302 is configured to send data information to be transferred from the source cluster to the monitoring organization;

[0069] An information processing module 303 is configured to obtain the data information to be transferred from the target cluster through the monitoring mechanism, create a target table for storing the data to be transferred, and send the target table information to the monitoring mechanism;

[0070] The data transfer module 304 is configured to enable the source cluster to obtain the target table information through the monitoring mechanism, and then send the data to be transferred to the target table.

[0071] According to one embodiment of the present invention, the association establishing module 301 may also be used to:

[0072] The source cluster and the target cluster are associated with each other by creating a first transient node on the monitoring mechanism.

[0073] According to another embodiment of the present invention, the information sending module 302 may also be used to:

[0074] The source cluster sends the data information to be transferred to the monitoring mechanism by means of a command parser command or calling an external interface.

[0075] According to another embodiment of the present invention, the apparatus 300 for inter-cluster data transfer may further include a node creation module (not shown in the figure) configured to: after the source cluster sends the data information to be transferred to the monitoring mechanism, the monitoring mechanism creates a second transient node for storing the data information to be transferred, wherein the data information to be transferred includes a target cluster identifier;

[0076] Furthermore, the information processing module 303 may also be used to:

[0077] The target cluster acquires the data information to be transferred by monitoring a second transient node related to the target cluster on the monitoring mechanism.

[0078] According to yet another embodiment of the present invention, the data transfer module 304 may also be used to:

[0079] The source cluster asynchronously sends the data to be transferred to the target table through remote procedure calls according to a mutual trust agreement between the source cluster and the target cluster.

[0080] According to an embodiment of the present invention, the monitoring mechanism monitors the status of the source cluster and the target cluster, as well as the data location transferred by the source cluster and the data location received by the target cluster. Moreover, during the data transfer process, if an abnormal node status occurs in the cluster, the monitoring mechanism notifies the cluster where the node status abnormality occurs to use other nodes for retry.

[0081] According to another embodiment of the present invention, the apparatus 300 for inter-cluster data transfer may further include a service switching module (not shown in the figure) configured to:

[0082] When the data transfer is completed, the source cluster deletes the data to be transferred and reports deletion completion information to the monitoring agency;

[0083] The target cluster obtains the deletion completion information reported by the source cluster through the monitoring mechanism, and provides services for the data to be transferred;

[0084] The association between the source cluster and the target cluster is disconnected by the monitoring mechanism.

[0085] According to the technical solution of the embodiment of the present invention, an association between a source cluster and a target cluster is established through a monitoring mechanism; the source cluster sends the data information to be transferred to the monitoring mechanism; the target cluster obtains the data information to be transferred through the monitoring mechanism, creates a target table for storing the data to be transferred, and sends the target table information to the monitoring mechanism; the source cluster obtains the target table information through the monitoring mechanism, and then sends the data to be transferred to the target table, thereby realizing intelligent transfer of data between clusters based on the monitoring mechanism without manual intervention, solving the problems of cumbersome and error-prone operations caused by manual operations, and greatly improving the processing efficiency of operation and maintenance personnel. At the same time, the monitoring mechanism can also monitor the cluster status and the status during the data transfer process, so as to facilitate timely detection of problems and retry, thereby ensuring that data transfer can be carried out safely and effectively.

[0086] Figure 4 An exemplary system architecture 400 is shown to which the method or apparatus for inter-cluster data transfer according to an embodiment of the present invention may be applied.

[0087] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0088] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Terminal devices 401, 402, and 403 can be installed with various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0089] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0090] Server 405 may be a server that provides various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 401, 402, and 403. The backend management server may analyze and process received data such as product information query requests, and feed back processing results (for example, target push information and product information—for example only) to the terminal device.

[0091] It should be noted that the method for inter-cluster data transfer provided by the embodiment of the present invention is generally executed by the server 405 . Accordingly, the apparatus for inter-cluster data transfer is generally disposed in the server 405 .

[0092] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0093] Reference below Figure 5 , which shows a schematic structural diagram of a computer system 500 of a terminal device or server suitable for implementing an embodiment of the present invention. Figure 5 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0094] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the system 500 are also stored in the RAM 503. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0095] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 508 as needed.

[0096] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from a removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are performed.

[0097] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be provided in a processor. For example, they may be described as: a processor including an association establishment module, an information sending module, an information processing module, and a data transfer module. The names of these units or modules do not, in certain circumstances, constitute limitations on the units or modules themselves. For example, the association establishment module may also be described as a "module for establishing an association between a source cluster and a target cluster through a monitoring mechanism."

[0100] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by a device, the device includes: establishing an association between a source cluster and a target cluster through a monitoring mechanism; the source cluster sends data information to be transferred to the monitoring mechanism; the target cluster obtains the data information to be transferred through the monitoring mechanism, creates a target table for storing the data to be transferred, and sends the target table information to the monitoring mechanism; the source cluster obtains the target table information through the monitoring mechanism, and then sends the data to be transferred to the target table.

[0101] According to the technical solution of the embodiment of the present invention, an association between a source cluster and a target cluster is established through a monitoring mechanism; the source cluster sends the data information to be transferred to the monitoring mechanism; the target cluster obtains the data information to be transferred through the monitoring mechanism, creates a target table for storing the data to be transferred, and sends the target table information to the monitoring mechanism; the source cluster obtains the target table information through the monitoring mechanism, and then sends the data to be transferred to the target table, thereby realizing intelligent transfer of data between clusters based on the monitoring mechanism without manual intervention, solving the problems of cumbersome and error-prone operations caused by manual operations, and greatly improving the processing efficiency of operation and maintenance personnel. At the same time, the monitoring mechanism can also monitor the cluster status and the status during the data transfer process, so as to facilitate timely detection of problems and retry, thereby ensuring that data transfer can be carried out safely and effectively.

[0102] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for transferring data between clusters, characterized in that: include: Establishing the association between the source cluster and the target cluster through the monitoring organization; The source cluster sends the data information to be transferred to the monitoring organization; The monitoring mechanism creates a second transient node for storing the data information to be transferred, wherein the data information to be transferred includes a target cluster identifier, wherein the second transient node is further used to record information during the transfer process, wherein the information during the transfer process includes the amount of data to be sent, the location of data sent by the source cluster, and the location of data received by the target cluster; The target cluster obtains the data information to be transferred through the monitoring mechanism, creates a target table for storing the data to be transferred, and sends the target table information to the monitoring mechanism; The source cluster obtains the target table information through the monitoring mechanism, and then sends the data to be transferred to the target table. The sending of the data to be transferred to the target table is achieved by sending the data to be transferred to a node of the target cluster, so that the node obtains the data to be transferred and stores it in the target table. Furthermore, the target cluster obtaining the data information to be transferred through the monitoring mechanism includes: the target cluster obtaining the data information to be transferred by monitoring a second transient node related to the target cluster on the monitoring mechanism.

2. The method according to claim 1, characterized in that Establishing an association between the source cluster and the target cluster through the monitoring mechanism includes: The source cluster and the target cluster are associated with each other by creating a first transient node on the monitoring mechanism.

3. The method according to claim 1, characterized in that The source cluster sending the data information to be transferred to the monitoring mechanism includes: The source cluster sends the data information to be transferred to the monitoring mechanism by means of a command parser command or calling an external interface.

4. The method according to claim 1, wherein The source cluster sending the data to be transferred to the target table includes: The source cluster asynchronously sends the data to be transferred to the target table through remote procedure calls according to a mutual trust agreement between the source cluster and the target cluster.

5. The method according to claim 1, wherein The monitoring mechanism monitors the status of the source cluster and the target cluster, as well as the data location transferred by the source cluster and the data location received by the target cluster. Moreover, during the data transfer process, if a node status abnormality occurs in the cluster, the monitoring mechanism notifies the cluster where the node status abnormality occurs to use other nodes for retry.

6. The method according to claim 1, characterized in that The method further comprises: When the data transfer is completed, the source cluster deletes the data to be transferred and reports deletion completion information to the monitoring agency; The target cluster obtains the deletion completion information reported by the source cluster through the monitoring mechanism, and provides services for the data to be transferred; The association between the source cluster and the target cluster is disconnected by the monitoring mechanism.

7. A device for transferring data between clusters, characterized in that: include: An association establishment module, used to establish an association between the source cluster and the target cluster through the monitoring mechanism; An information sending module, configured to send data information to be transferred from the source cluster to the monitoring organization; a node creation module, configured to create, by the monitoring mechanism, a second transient node for storing the data information to be transferred, the data information to be transferred including a target cluster identifier, wherein the second transient node is further configured to record information during the transfer process, the information during the transfer process including the amount of data to be sent, the location of data sent by the source cluster, and the location of data received by the target cluster; an information processing module, configured to obtain, by the target cluster, the data information to be transferred through the monitoring mechanism, create a target table for storing the data to be transferred, and send the target table information to the monitoring mechanism; a data transfer module, configured to obtain the target table information by the source cluster through the monitoring mechanism, and then send the data to be transferred to the target table, wherein the sending of the data to be transferred to the target table is achieved by sending the data to be transferred to a node of the target cluster, so that the node obtains the data to be transferred and stores it in the target table; Furthermore, the information processing module is further configured to enable the target cluster to acquire the data information to be transferred by monitoring a second transient node related to the target cluster on the monitoring mechanism.

8. An electronic device for inter-cluster data transfer, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images