Data replication method, apparatus, system, and storage medium

Abstract

The application provides a data replication method, device, system and storage medium, and belongs to the technical fields of big data and data processing. The method comprises the following steps: generating replication configuration parameters and a node opening instruction, determining source table data of source data, determining source table metadata, establishing a data transmission channel between each node of a source cluster end and each node of a target cluster end, determining a replication mapping relationship between the nodes of the source cluster end and the nodes of the target cluster end, splitting the source table data, generating target table metadata according to the source table metadata, and generating initial table data according to the target table metadata; and sending the split source table data from the multiple nodes of the source cluster end to the multiple nodes of the target cluster end through the data transmission channel, so that the target cluster end imports the source table data into the initial table data according to the target table metadata to generate target table data. The application can improve the efficiency of data replication.

Description

Technical Field

[0001] This application relates to the fields of big data and data processing technology, and in particular to a data replication method, apparatus, system and storage medium. Background Technology

[0002] With the ever-increasing volume of enterprise-level big data, MPP (Massively Parallel Processing) database clusters are widely used in cloud data warehouses. To meet users' needs for batch queries and achieve high availability between master and slave database clusters, data replication between clusters is often required.

[0003] Currently, in existing technologies, when copying data between clusters, the source data terminal parses the configuration file input by the staff on the source cluster terminal to obtain the parsing results containing filtering conditions. Then, it uses database query commands to read the target data from the database on the source cluster terminal according to the parsing results of the filtering conditions, and then copies the target data to the target database on the target cluster terminal.

[0004] However, the inventors have discovered that the existing technology has at least the following technical problems: when it is necessary to perform full copying of database tables or data copying or migration between entire databases, there is still a problem of low copying efficiency. Summary of the Invention

[0005] This application provides a data replication method, apparatus, system, and storage medium that can improve data replication efficiency when performing full copying of database tables or data replication or migration at the whole database level between clusters.

[0006] In a first aspect, this application provides a data copying method, comprising:

[0007] In response to a data replication request, generate replication configuration parameters and node activation instructions;

[0008] Based on the replication configuration parameter information, determine the source table data of the source data; based on the source database system tables, determine the source table metadata;

[0009] According to the node activation command, a data transmission channel is established between each node of the source cluster and each node of the target cluster.

[0010] Receive the target database system table sent by the target cluster terminal;

[0011] Based on the target database system table and the source database system table, determine the replication mapping relationship between the nodes of the source cluster and the nodes of the target cluster;

[0012] Based on the replication mapping relationship, the source table data is split;

[0013] The source table metadata is sent to the target cluster, so that the target cluster generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata.

[0014] The split source table data is sent from multiple nodes in the source cluster to multiple nodes in the target cluster through the data transmission channel, so that the target cluster can import the source table data into the initial table data according to the target table metadata to generate the target table data.

[0015] In one possible implementation, establishing a data transmission channel between each node in the source cluster and each node in the target cluster according to the node activation instruction includes: opening each node in the source cluster according to the node activation instruction; sending the node activation instruction to the target cluster so that the target cluster opens each node corresponding to all nodes in the source cluster according to the node activation instruction; determining database superuser whitelist information according to the node activation instruction; sending the database superuser whitelist information to the target cluster so that the target cluster determines network access correspondences according to the database superuser whitelist information; receiving the network access correspondences sent by the target cluster, and establishing a data transmission channel between each node in the source cluster and each node in the target cluster according to the network access correspondences.

[0016] In one possible implementation, the source table metadata includes metadata database schema definition language (DDL) information. Accordingly, sending the source table metadata to the target cluster so that the target cluster generates target table metadata based on the source table metadata and generates initial table data based on the target table metadata includes: sending the metadata DDL information to the target cluster so that the target cluster executes a pre-stored synchronization tool based on the database superuser whitelist information to synchronize the source table metadata from the source cluster to the target cluster, and generates target table metadata based on the metadata DDL information and generates initial table data based on the target table metadata.

[0017] In one possible implementation, determining the replication mapping relationship between nodes in the source cluster and nodes in the target cluster based on the target database system table and the source database system table includes: determining the Internet Protocol (IP) information of each node in the source cluster based on the source database system table; determining the IP information of each node in the target cluster based on the target database system table; and determining the replication mapping relationship between each node in the source cluster and each node in the target cluster based on the IP information of each node in the source cluster and the IP information of each node in the target cluster.

[0018] In one possible implementation, the response to the data copy request further includes generating statistical information about the source data.

[0019] In one possible implementation, the statistical information of the source data includes the number of records in the source table. Correspondingly, after the split source table data is sent from multiple nodes in the source cluster to multiple nodes in the target cluster via the data transmission channel, so that the target cluster imports the source table data into the initial table data based on the target table metadata to generate the target table data, the process further includes: receiving the target record count sent by the target cluster, where the target record count is generated when the target cluster imports the source table data into the initial table data; performing a data comparison between the target record count and the source table record count to generate a record count comparison result; if the record count comparison result meets a preset error condition, returning to the step of generating replication configuration parameters and node activation instructions in response to the data replication request; and if the record count comparison result meets a preset success condition, returning a data replication success result.

[0020] In one possible implementation, the statistical information of the source data includes the number of records in the source table. Accordingly, after sending the source table metadata to the target cluster so that the target cluster generates target table metadata based on the source table metadata and generates initial table data based on the target table metadata, the method further includes: if it is determined that the number of records in the source table meets a preset threshold condition, then according to the replication mapping relationship, the source table data is sent from one of the nodes of the source cluster to the target cluster through the data transmission channel so that the target cluster imports the source table data into the initial table data based on the target table metadata and generates the target table data.

[0021] In one possible implementation, after sending the source table metadata to the target cluster so that the target cluster generates target table metadata based on the source table metadata and generates initial table data based on the target table metadata, the method further includes: sending a same-table query instruction to the target cluster so that the target cluster determines the target data table with the same source table metadata based on the same-table query instruction and deletes the target data table.

[0022] In one possible implementation, before splitting the source table data according to the replication mapping relationship, the method further includes: if it is determined that the number of nodes on the source cluster side is not equal to the number of nodes on the target cluster side, then the source table data is redistributed to generate redistributed source table data.

[0023] In one possible implementation, after the split source table data is sent from multiple nodes of the source cluster to multiple nodes of the target cluster via the data transmission channel, so that the target cluster imports the source table data into the initial table data according to the target table metadata and generates the target table data, the method further includes: sending a statistical information collection instruction to the target cluster so that the target cluster collects statistical information of the target table data according to the statistical information collection instruction.

[0024] Secondly, this application also provides a data replication method applied to a target cluster, including:

[0025] The system receives a node activation instruction sent by the source cluster, wherein the node activation instruction is generated by the source cluster in response to a data replication request, and the replication configuration parameters are used to instruct the source cluster to determine the source table data of the source data based on the replication configuration parameter information; and to determine the source table metadata based on the source database system table.

[0026] According to the node activation command, a data transmission channel is established between each node of the source cluster and each node of the target cluster.

[0027] The target database system table is sent to the source cluster end, so that the source cluster end determines the replication mapping relationship between the nodes of the source cluster end and the nodes of the target cluster end based on the target database system table and the source database system table, and splits the source table data according to the replication mapping relationship.

[0028] Receive source table metadata sent by the source cluster end, generate target table metadata based on the source table metadata, and generate initial table data based on the target table metadata;

[0029] Receive the split source table data sent by the source cluster end through the data transmission channel, wherein the split source table data is sent from multiple nodes of the source cluster end to multiple nodes of the target cluster end;

[0030] Based on the target table metadata, the source table data is imported into the initial table data to generate the target table data.

[0031] In one possible implementation, establishing a data transmission channel between each node of the source cluster and each node of the target cluster according to the node activation instruction includes: receiving a node activation instruction sent by the source cluster, wherein the node activation instruction is used by the source cluster to activate each node of the source cluster and determine database super user whitelist information; activating each node corresponding to all nodes of the source cluster according to the node activation instruction; receiving the database super user whitelist information sent by the source cluster and determining network access correspondence based on the database super user whitelist information; and sending the network access correspondence to the source cluster, so that the source cluster establishes a data transmission channel between each node of the source cluster and each node of the target cluster according to the network access correspondence.

[0032] In one possible implementation, the source table metadata includes metadata database schema definition language (DDL) information. Correspondingly, the process of receiving the source table metadata sent by the source cluster, generating target table metadata based on the source table metadata, and generating initial table data based on the target table metadata includes: receiving the metadata DDL information sent by the source cluster, executing a pre-stored synchronization tool based on the database superuser whitelist information to synchronize the source table metadata from the source cluster to the target cluster; generating target table metadata based on the metadata DDL information; and generating initial table data based on the target table metadata.

[0033] In one possible implementation, the source cluster terminal, in response to the data replication request, also generates statistical information about the source data, including the number of records in the source table. Correspondingly, after importing the source table data into the initial table data based on the target table metadata to generate the target table data, the process further includes: generating a target number of records when importing the source table data into the initial table data; sending the target number of records to the source cluster terminal so that the source cluster terminal performs a data comparison process between the target number of records and the number of records in the source table, generating a record number comparison result; and if the record number comparison result is determined to meet a preset error condition, returning to the step of generating replication configuration parameters and node activation instructions in response to the data replication request; and if the record number comparison result is determined to meet a preset success condition, returning a data replication success result.

[0034] In one possible implementation, after importing the source table data into the initial table data according to the target table metadata to generate the target table data, the method further includes: when the source cluster determines that the number of records in the source table meets a preset threshold condition, receiving the source table data sent by one of the nodes of the source cluster through the data transmission channel; and importing the source table data into the initial table data according to the target table metadata to generate the target table data.

[0035] In one possible implementation, after generating target table metadata based on the source table metadata and generating initial table data based on the target table metadata, the method further includes: receiving a same-table query instruction sent by the source cluster; determining the target data table with the same source table metadata based on the same-table query instruction, and deleting the target data table.

[0036] In one possible implementation, after importing the source table data into the initial table data according to the target table metadata to generate the target table data, the method further includes: receiving a statistical information instruction sent by the source cluster end, and collecting statistical information of the target table data according to the statistical information instruction.

[0037] Thirdly, this application provides a data replication apparatus applied at the source cluster end, comprising:

[0038] The request and response module is used to respond to data replication requests and generate replication configuration parameters and node activation instructions;

[0039] The source data processing module is used to determine the source table data of the source data based on the replication configuration parameter information; and to determine the source table metadata based on the source database system table.

[0040] The first transmission channel establishment module is used to establish a data transmission channel between each node of the source cluster and each node of the target cluster according to the node activation instruction.

[0041] The first receiving module is used to receive the target database system table sent by the target cluster terminal;

[0042] The replication mapping relationship determination module is used to determine the replication mapping relationship between the nodes of the source cluster and the nodes of the target cluster based on the target database system table and the source database system table.

[0043] The data splitting module is used to split the source table data according to the replication mapping relationship;

[0044] The first sending module is used to send the source table metadata to the target cluster end, so that the target cluster end generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata;

[0045] The first sending module is further configured to send the split source table data from multiple nodes of the source cluster to multiple nodes of the target cluster through the data transmission channel, so that the target cluster can import the source table data into the initial table data and generate the target table data according to the target table metadata.

[0046] Fourthly, this application also provides a data replication apparatus, applied to a target cluster, comprising:

[0047] The second receiving module is used to receive a node activation instruction sent by the source cluster end, wherein the node activation instruction is generated by the source cluster end in response to the data replication request, and the replication configuration parameters are used to instruct the source cluster end to determine the source table data of the source data according to the replication configuration parameter information; and to determine the source table metadata according to the source database system table.

[0048] The second transmission channel establishment module is used to establish a data transmission channel between each node of the source cluster and each node of the target cluster according to the node activation instruction.

[0049] The second sending module is used to send the target database system table to the source cluster end, so that the source cluster end can determine the replication mapping relationship between the nodes of the source cluster end and the nodes of the target cluster end based on the target database system table and the source database system table, and split the source table data according to the replication mapping relationship.

[0050] The second receiving module is further configured to receive source table metadata sent by the source cluster end, generate target table metadata based on the source table metadata, and generate initial table data based on the target table metadata;

[0051] The second receiving module is further configured to receive the split source table data sent by the source cluster end through the data transmission channel, wherein the split source table data is sent from multiple nodes of the source cluster end to multiple nodes of the target cluster end;

[0052] The data replication module is used to import source table data into initial table data based on the target table metadata, thereby generating target table data.

[0053] Fifthly, this application provides a server system, including: at least one processor and a memory;

[0054] The memory stores computer-executed instructions;

[0055] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform a data copying method as described in the first or second aspect.

[0056] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the data copying method as described in the first or second aspect.

[0057] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data copying method as described in the first or second aspect.

[0058] This application provides a data replication method, apparatus, system, and storage medium. In response to a data replication request, it generates replication configuration parameters and node activation instructions. Based on the replication configuration parameters, it determines the source table data and then the source table metadata based on the source database system table. The source table metadata is then sent to the target cluster, enabling the target cluster to generate target table metadata and initial table data, thus completing the replication of the source table metadata. Upon receiving the target database system table, it determines the replication mapping relationship between nodes in the source cluster and nodes in the target cluster based on the target and source database system tables, and then splits the source table data according to this mapping relationship. Finally, the split source table data is sent from multiple nodes in the source cluster to multiple nodes in the target cluster via a data transmission channel. This allows the target cluster to import the source table data into the initial table data and generate the target table data based on the target table metadata. This method leverages the data processing and communication bandwidth capabilities of multiple nodes working simultaneously across the cluster, improving the replication efficiency of source data between the source and target clusters. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram illustrating an application scenario of the data copying method provided in the embodiments of this application;

[0061] Figure 2A schematic flowchart illustrating a data copying method provided in one embodiment of this application;

[0062] Figure 3 A flowchart illustrating a data copying method provided in another embodiment of this application;

[0063] Figure 4 A schematic diagram of the interaction flow of the data copying method provided in the embodiments of this application;

[0064] Figure 5 A schematic diagram of a data copying device provided in an embodiment of this application;

[0065] Figure 6 This is a schematic diagram of another data copying device provided in an embodiment of this application;

[0066] Figure 7 This is a schematic diagram of the hardware structure of the server system provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] Currently, with the ever-increasing volume of enterprise-level big data, MPP (Massively Parallel Processing) database clusters are widely used in cloud data warehouses. To meet users' batch query needs and achieve high availability between database clusters, data replication between clusters is often required. In existing technologies, when replicating data between clusters, the source database parses the configuration file input by the user, obtaining a parsed result containing filtering conditions. Then, database query commands are used to read the target data from the source cluster's database based on the parsed results of the filtering conditions, and the target data is copied to the target database in the target cluster. The inventors have discovered that when performing full copying of tables across a large number of databases or data replication or migration at the entire database level, this method, which only filters source data that meets the filtering conditions and only unloads data from the cluster master node for data replication, still suffers from low replication efficiency.

[0069] To address the aforementioned technical problems, this application provides the following technical concept: the source data to be replicated is divided into two parts: source table data and source table metadata. The source table metadata is replicated from the source cluster to the target cluster using nodes on both the source and target cluster sides. Then, the target cluster generates target table metadata based on the source table metadata and constructs initial table data identical to the source table data in terms of table structure and user permissions. The source table data leverages the data processing and bandwidth communication capabilities of multiple nodes on both the source and target cluster sides to improve the transmission efficiency of the source table metadata from the source cluster to the target cluster, thereby increasing the replication efficiency of the source table data.

[0070] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the data copying method provided in the embodiments of this application, such as... Figure 1 As shown, it includes a source cluster terminal 101 and a target cluster terminal 102. Both the source cluster terminal 101 and the target cluster terminal 102 can consist of multiple nodes, including one master node and multiple data nodes. The master node and data nodes can be computers or servers. When data replication is required, staff can configure parameters and install data replication tools on either the source cluster terminal 101 or the target cluster terminal 102. For example, when a node in the source cluster terminal 101 receives a user-triggered data replication request, the source cluster terminal 101 responds to the request by instructing itself to copy the source data to be replicated to the target cluster terminal 102.

[0071] like Figure 1 In the application scenario shown, the source cluster 101 can be a GP (Greenplum Distributed Database) cluster or an MPP (Massively Parallel Processing) cluster, and the target cluster 102 can also be a GP cluster or an MPP cluster.

[0072] Figure 2 This is a flowchart illustrating a data copying method provided in one embodiment of this application. The execution entity of this embodiment can be... Figure 1 Any node in the source cluster 101 of the illustrated embodiment can be a server or other computer-related devices; this embodiment is not particularly limited in this respect. Figure 2 As shown, the data copying method includes:

[0073] S201: In response to a data replication request, generate replication configuration parameters and node activation instructions.

[0074] In this embodiment, a data replication request can be triggered by any node in the source cluster. The replication configuration parameters can include a list of tables to be replicated, relevant information from the target cluster, and table mapping relationships, as well as other parameters required for data replication. The node activation command is an instruction to activate the corresponding nodes on both the source and target clusters.

[0075] S202: Based on the replication configuration parameter information, determine the source table data of the source data; based on the source database system tables, determine the source table metadata.

[0076] In this embodiment, the source data can be an entire database, and database replication can include the replication of both table data and table metadata. The source table data includes the source database tables and the data within those tables to be exported and replicated to the target cluster. The source database system tables are automatically generated by the system when storing data on the source cluster side, used to store system information and the corresponding data storage location in the database. The source table metadata can include statistical information, table structure definition information, table usage permissions, and view dependencies. For example, based on the list of tables to be replicated in the replication configuration parameters, the source data is queried to determine the source table data.

[0077] S203: Based on the node activation command, establish a data transmission channel between each node in the source cluster and each node in the target cluster.

[0078] In this embodiment, the node activation command is used to activate multiple nodes on the source cluster side and multiple nodes on the target cluster side, enabling data transmission between each node on the source cluster side and each node on the target cluster side. A data transmission channel refers to the pathway through which signals can be transmitted between two nodes.

[0079] In an optional embodiment of this application, step S203 includes:

[0080] S203a: Based on the node activation command, enable each node on the source cluster; send the node activation command to the target cluster so that the target cluster enables each node corresponding to all nodes on the source cluster according to the node activation command; determine the database super user whitelist information based on the node activation command.

[0081] S203b: Send the database superuser whitelist information to the target cluster so that the target cluster can determine the network access correspondence based on the database superuser whitelist information.

[0082] S203c: Receives the network access correspondence sent by the target cluster end, and establishes a data transmission channel between each node of the source cluster end and each node of the target cluster end based on the network access correspondence.

[0083] In this embodiment, "open node" refers to granting access permissions to the node server. "Database superuser whitelist information" refers to user permission information; once the database superuser whitelist is enabled, users on the source cluster can log in without a password on the target cluster. "Network access mapping relationship" refers to the connection or access path information between two corresponding nodes; for example, the network mapping relationship between a web browser and a website is the website's URL.

[0084] S204: Receive the target database system table sent by the target cluster.

[0085] In this embodiment, the target database system table is a table automatically generated by the system when storing data on the target cluster side. It is used to store system information and information such as the corresponding data storage location in the database.

[0086] S205: Based on the target database system tables and the source database system tables, determine the replication mapping relationship between the nodes on the source cluster end and the nodes on the target cluster end.

[0087] In this embodiment, the replication mapping relationship is a correspondence between two objects. For example, in this embodiment, the two objects are a correspondence between a node on the source cluster and a node on the target cluster.

[0088] In an optional embodiment of this application, step S205 includes:

[0089] S205a: Determine the Internet Protocol information for each node in the source cluster based on the source database system table.

[0090] S205b: Determine the Internet Protocol information for each node in the target cluster based on the target database system table.

[0091] S205c: Based on the Internet Protocol information of each node in the source cluster and the Internet Protocol information of each node in the target cluster, determine the replication mapping relationship between each node in the source cluster and each node in the target cluster.

[0092] In this embodiment, the Internet Protocol (IP) information of a node is the IP address of its physical network interface card (NIC). All nodes' IP information represents a real, existing physical network. The IP information of each node in the source cluster indicates that when a node outside the source cluster accesses a node or service within the source cluster, it must communicate through the physical network of this source cluster. Similarly, the IP information of each node in the target cluster indicates that when a node outside the target cluster accesses a node or service within the target cluster, it must communicate through the physical network of the target cluster.

[0093] In this embodiment, the replication mapping relationship between each node in the source cluster and each node in the target cluster is specifically manifested as follows: for one node in the source cluster, there is a corresponding node in the target cluster, and the Internet Protocol addresses of the two nodes correspond.

[0094] S206: Split the source table data according to the replication mapping relationship.

[0095] In this embodiment, splitting the source table data can be achieved by dividing the data into multiple parts and exporting them from multiple nodes. For example, the cluster helper tool can be used to export the source table data from the source cluster.

[0096] S207: Send the source table metadata to the target cluster so that the target cluster can generate the target table metadata based on the source table metadata, and generate the initial table data based on the target table metadata.

[0097] In this embodiment, the source table metadata is the metadata generated when the source cluster queries the source database system tables, and the target table metadata is the source table metadata copied from the source cluster to the target cluster. The initial table data is a data table that has not yet been stored with data, and this data table has the same table structure, user permissions, and other basic table data information as the source data.

[0098] In an optional embodiment of this application, the source table metadata includes metadata database schema definition language (DDL) information. Accordingly, step S207 includes:

[0099] The metadata DDL information is sent to the target cluster so that the target cluster can execute the pre-stored synchronization tool based on the database super user whitelist information. This tool synchronizes the source table metadata from the source cluster to the target cluster, generates the target table metadata based on the metadata DDL information, and generates the initial table data based on the target table metadata.

[0100] In this embodiment, the sending of metadata DDL information can be performed between the master node on the source cluster side and the master node on the target cluster side.

[0101] In this embodiment, metadata DDL information refers to operation commands related to defining metadata, such as commands to create, modify, and delete metadata. In this embodiment, the pre-stored synchronous execution tool can be a replication tool, such as the hashcopy tool.

[0102] S208: The split source table data is sent from multiple nodes on the source cluster to multiple nodes on the target cluster through the data transmission channel, so that the target cluster can import the source table data into the initial table data according to the target table metadata and generate the target table data.

[0103] In this embodiment, step S208 can be completed by executing multiple replication tools simultaneously on the source cluster side, for example, by executing the hashcopy tool simultaneously on multiple nodes on the source cluster side.

[0104] In summary, the data replication method provided in this embodiment generates replication configuration parameters and node activation instructions in response to a data replication request. Based on the replication configuration parameters, it determines the source table data and the source table metadata. Then, it sends the source table metadata to the target cluster, enabling the target cluster to generate target table metadata and initial table data, thus completing the replication of the source table metadata. Upon receiving the target database system table, it determines the replication mapping relationship between nodes in the source and target clusters based on the target and source database system tables, and splits the source table data according to this mapping relationship. Finally, it sends the split source table data from multiple nodes in the source cluster to multiple nodes in the target cluster via a data transmission channel, allowing the target cluster to import the source table data into the initial table data and generate the target table data. This method leverages the data processing and communication bandwidth capabilities of multiple nodes working simultaneously across the cluster, improving the replication efficiency of source data between the source and target clusters.

[0105] Based on the above embodiments, as an optional embodiment of this application, step S201, after responding to the data copying request, further includes:

[0106] Step A: Generate statistical information for the source data.

[0107] In this embodiment, the statistical information of the source data can be the number of records such as the number of databases to be copied and the number of source data.

[0108] Based on the above embodiments, in an optional embodiment of this application, the statistical information of the source data includes the number of records in the source table. Accordingly, after step S208, the method further includes:

[0109] Step B: Receive the target record count sent by the target cluster. The target record count is generated when the target cluster imports the source table data into the initial table data.

[0110] Step C: Compare the target number of records with the number of records in the source table to generate a record count comparison result.

[0111] In this embodiment, the data comparison processing is a record count consistency comparison processing, and the record count comparison result may include record count consistency or record count inconsistency.

[0112] Step D: If the record count comparison result meets the preset error reporting conditions, return to the step of responding to the data replication request and generating replication configuration parameters and node activation instructions.

[0113] Step E: If the record count comparison result meets the preset success conditions, then return a successful data copy result.

[0114] In this embodiment, the preset error condition is that the number of records is inconsistent. When the number of records is inconsistent, it indicates that the data copying has failed, and it is necessary to continue from step S201 to step S208 until the data comparison result shows that the number of records is consistent.

[0115] In summary, the data replication method provided in this embodiment compares the target number of records with the number of records in the source table to generate a record count comparison result. Based on different record count comparison results, corresponding processing measures are selected until the data replication is successful, thus avoiding data replication failure or data omission.

[0116] Based on the above embodiments, as an optional embodiment of this application, the statistical information of the source data includes the number of records in the source table. Accordingly, after step S207, the following is also included:

[0117] Step F: If the number of records in the source table meets the preset threshold condition, then according to the replication mapping relationship, the source table data is sent from one of the nodes in the source cluster to the target cluster through the data transmission channel, so that the target cluster can import the source table data into the initial table data according to the target table metadata and generate the target table data.

[0118] In this embodiment, the preset threshold condition is a manually set numerical condition, such as 1 million. When the number of records in the source table meets the preset threshold condition, it indicates that the data processing capacity and bandwidth communication capacity of one node in the source cluster and one node in the target cluster are sufficient to meet the data replication requirements.

[0119] In this embodiment, if the number of records in the source table does not meet the preset threshold condition, the source table data is sent using step S208.

[0120] In summary, the data replication method provided in this embodiment determines whether to use one node or multiple nodes to send source table data by judging whether the number of records in the source table meets the preset threshold condition, thereby saving data processing data and bandwidth communication resources on both the source cluster end and the target cluster end.

[0121] Based on the above embodiments, in an optional embodiment of this application, after step S207, the method further includes:

[0122] Step G: Send the same-table query command to the target cluster end, so that the target cluster end can determine the target end data table with the same source table metadata according to the same-table query command, and delete the target end data table.

[0123] In this embodiment, the same table query instruction is an instruction to the target cluster to query whether there is a database or data table with the same data. The target data table is deleted before step S208 is performed.

[0124] In summary, the data replication method provided in this embodiment improves data replication efficiency by sending a same-table query command to the target cluster, causing the target cluster to delete the target data table that has the same metadata as the source table, thus avoiding the insertion of duplicate data.

[0125] Based on the above embodiments, in an optional embodiment of this application, before step S206, the following step is further included:

[0126] Step H: If the number of nodes in the source cluster is not equal to the number of nodes in the target cluster, the source table data is redistributed to generate redistributed source table data.

[0127] In this embodiment, the redistribution process is characterized by a discrepancy between the number of nodes in the source cluster and the number of nodes in the target cluster, indicating a mismatch in cluster size between the two. In this embodiment, redistribution refers to redistributing the existing data of the cluster to new nodes, appropriately reducing the amount of data in the old nodes.

[0128] In summary, the data replication method provided in this embodiment improves data replication efficiency by redistributing the source table data, enabling data to be transmitted more quickly from multiple nodes in the source cluster to multiple nodes in the target cluster.

[0129] Based on the above embodiments, as an optional embodiment of this application, after step S208, the method further includes:

[0130] Step 1: Send the statistical information collection instruction to the target cluster end so that the target cluster end can collect statistical information of the target table data according to the statistical information collection instruction.

[0131] In this embodiment, the statistical information collection instruction is an instruction to the target cluster to collect statistical information of the target table data. The statistical information of the target table data may include the amount of data in the target table.

[0132] In summary, the data replication method provided in this embodiment facilitates subsequent use and retrieval of target table data or execution plans by collecting statistical information of the target table data.

[0133] refer to Figure 3 , Figure 3This is a flowchart illustrating a data copying method provided in another embodiment of this application. The executing entity in this embodiment can be any type of server, or... Figure 1 The target cluster shown is not specifically limited in this embodiment. Figure 3 As shown, the method includes:

[0134] S301: Receive the node activation instruction sent by the source cluster end. The node activation instruction is generated by the source cluster end in response to the data replication request, which includes replication configuration parameters and the node activation instruction. The replication configuration parameters are used to instruct the source cluster end to determine the source table data of the source data based on the replication configuration parameter information and to determine the source table metadata based on the source database system table.

[0135] S302: Based on the node activation command, establish a data transmission channel between each node in the source cluster and each node in the target cluster.

[0136] Specifically, in an optional embodiment of this application, step S302 includes:

[0137] S302a: Receive node activation instructions sent by the source cluster end. The node activation instructions are used by the source cluster end to enable each node of the source cluster end and determine the database super user whitelist information.

[0138] S302b: Enables each node corresponding to all nodes in the source cluster according to the node activation command.

[0139] S302c: Receives the database super user whitelist information sent by the source cluster end, and determines the network access correspondence based on the database super user whitelist information.

[0140] S302d: Sends network access relationships to the source cluster end, so that the source cluster end can establish a data transmission channel between each node of the source cluster end and each node of the target cluster end according to the network access correspondence.

[0141] In this embodiment, step S302 and Figure 2 The implementation principle and technical effect of step S203 in the illustrated embodiment are similar, so they will not be described again here.

[0142] S303: Send the target database system table to the source cluster end so that the source cluster end can determine the replication mapping relationship between the nodes of the source cluster end and the nodes of the target cluster end based on the target database system table and the source database system table, and split the source table data according to the replication mapping relationship.

[0143] In this embodiment, the target database system table is a table automatically generated by the system when storing data on the target cluster side. It is used to store system information and information such as the corresponding data storage location in the database.

[0144] S304: Receives source table metadata sent by the source cluster, generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata.

[0145] Specifically, in an optional embodiment of this application, the source table metadata includes metadata database schema definition language (DDL) information, and correspondingly, step S304 includes:

[0146] S304a: Receives metadata DDL information sent from the source cluster and executes the pre-stored synchronization tool according to the database super user whitelist information to synchronize the source table metadata from the source cluster to the target cluster.

[0147] S304b: Generate target table metadata based on the metadata DDL information, and generate initial table data based on the target table metadata.

[0148] In this embodiment, the target table metadata is the same as the source table metadata. Taking a data table as an example, the target table metadata may include basic table parameter information such as the number of rows, the number of columns, and the table header name. The initial table data is a data table without imported data.

[0149] S305: Receives split source table data sent by the source cluster end through the data transmission channel, wherein the split source table data is sent from multiple nodes of the source cluster end to multiple nodes of the target cluster end.

[0150] S306: Based on the target table metadata, import the source table data into the initial table data to generate the target table data.

[0151] In summary, the data replication method provided in this embodiment, when replicating source data from a source cluster to a target cluster, establishes data transmission channels between multiple nodes in the source cluster and multiple nodes in the target cluster. Multiple nodes simultaneously receive source table data sent from multiple nodes in the source cluster. Based on source table metadata sent from one node in the source cluster, target table metadata is first generated, and then initial table data is generated based on the target table metadata. Finally, the source table data sent from multiple nodes in the source cluster is imported into the initial table data based on the target table metadata to generate the target table data. This method leverages the data processing capabilities and bandwidth communication capabilities between multiple nodes to improve data replication efficiency when dealing with large amounts of data.

[0152] Based on the above embodiments, as an optional embodiment of this application, the source cluster end also generates statistical information of the source data in response to the data replication request. The statistical information of the source data includes the number of records in the source table. Accordingly, after step S306, the following is also included:

[0153] Step a: When importing data from the source table into the initial table, generate the target number of records.

[0154] Step b: Send the target number of records to the source cluster so that the source cluster can compare the target number of records with the number of records in the source table, generate a record comparison result, and if the record comparison result meets the preset error conditions, return to the step of responding to the data replication request, generating replication configuration parameters and node activation instructions, and if the record comparison result meets the preset success conditions, return a data replication success result.

[0155] Based on the above embodiments, in an optional embodiment of this application, after step S306, the method further includes:

[0156] Step c: When the source cluster determines that the number of records in the source table meets the preset threshold condition, it receives the source table data sent by one of the nodes in the source cluster through the data transmission channel.

[0157] Step d: Based on the target table metadata, import the source table data into the initial table data to generate the target table data.

[0158] In this embodiment, steps c to d are... Figure 2 The implementation process and technical effect of step F in the illustrated embodiment are similar, so they will not be described again here.

[0159] Based on the above embodiments, in an optional embodiment of this application, after step S306, the method further includes:

[0160] Step g: Receive the same table query command sent by the source cluster.

[0161] Step h: Based on the same table query instruction, identify the target data table with the same metadata as the source table, and delete the target data table.

[0162] In this embodiment, steps g to h are... Figure 2 The implementation principle and technical effect of step G in the illustrated embodiment are similar, so they will not be repeated here.

[0163] In an optional embodiment of this application, after step S306, the method further includes:

[0164] Step i: Receive the statistical information instruction sent by the source cluster end, and collect the statistical information of the target table data according to the statistical information instruction.

[0165] In this embodiment, the statistical information instruction is used to instruct the target cluster to collect statistical information during the process of generating target table data.

[0166] In summary, the data replication method provided in this embodiment facilitates the subsequent use of the target table data by collecting statistical information of the target table data on the target cluster side.

[0167] refer to Figure 4 , Figure 4 This is a schematic diagram of the interaction flow of the data copying method provided in the embodiments of this application. The data copying method includes:

[0168] S401: The source cluster responds to the data replication request by generating replication configuration parameters and node activation instructions.

[0169] S402: The source cluster determines the source table data based on the replication configuration parameters, and determines the source table metadata based on the source database system tables.

[0170] S403: The source cluster end establishes a data transmission channel between each node of the source cluster end and each node of the target cluster end according to the node activation command.

[0171] S404: The target cluster sends the target database system table to the source cluster.

[0172] S405: The source cluster determines the replication mapping relationship between the nodes of the source cluster and the nodes of the target cluster based on the target database system tables and the source database system tables.

[0173] S406: The source cluster splits the source table data according to the replication mapping relationship.

[0174] S407: The source cluster sends source table data to the target cluster.

[0175] S408: The target cluster generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata.

[0176] S409: Multiple nodes on the source cluster send the split source table data to multiple nodes on the target cluster through the data transmission channel.

[0177] S4010: The target cluster imports the source table data into the initial table data to generate the target table data.

[0178] In summary, the data replication method provided in this embodiment can improve data replication efficiency by utilizing multiple nodes on the source cluster and multiple nodes on the target cluster.

[0179] refer to Figure 5 , Figure 5 This is a schematic diagram of a data copying device provided in an embodiment of this application. Figure 5As shown, the data copying device 50 includes: a request response module 501, a source data processing module 502, a first transmission channel establishment module 503, a first receiving module 504, a copy mapping relationship determination module 505, a data splitting module 506, and a first sending module 507.

[0180] Specifically, the request-response module 501 is used to respond to data replication requests by generating replication configuration parameters and node activation instructions.

[0181] The source data processing module 502 is used to determine the source table data based on the replication configuration parameter information and to determine the source table metadata based on the source database system tables.

[0182] The first transmission channel establishment module 503 is used to establish a data transmission channel between each node of the source cluster and each node of the target cluster according to the node activation instruction.

[0183] The first receiving module 504 is used to receive the target database system table sent by the target cluster end.

[0184] The replication mapping relationship determination module 505 is used to determine the replication mapping relationship between nodes on the source cluster end and nodes on the target cluster end based on the target database system table and the source database system table.

[0185] The data splitting module 506 is used to split the source table data according to the replication mapping relationship.

[0186] The first sending module 507 is used to send the source table metadata to the target cluster end, so that the target cluster end can generate the target table metadata based on the source table metadata, and generate the initial table data based on the target table metadata.

[0187] The first sending module 507 is also used to send the split source table data from multiple nodes on the source cluster end to multiple nodes on the target cluster end through the data transmission channel, so that the target cluster end can import the source table data into the initial table data according to the target table metadata and generate the target table data.

[0188] In an optional embodiment of this application, the first transmission channel establishment module 503 is specifically used for:

[0189] Based on the node activation command, each node on the source cluster is enabled; the node activation command is sent to the target cluster so that the target cluster enables each node corresponding to all nodes on the source cluster according to the node activation command; based on the node activation command, the database superuser whitelist information is determined. The database superuser whitelist information is sent to the target cluster so that the target cluster determines the network access mapping relationship based on the database superuser whitelist information. The network access mapping relationship sent by the target cluster is received, and a data transmission channel is established between each node on the source cluster and each node on the target cluster based on the network access mapping relationship.

[0190] In an optional embodiment of this application, the source table metadata includes metadata database schema definition language (DDL) information. The first sending module 507 is specifically used to send the metadata DDL information to the target cluster end, so that the target cluster end executes the pre-stored synchronization execution tool according to the database super user whitelist information, synchronizes the source table metadata from the source cluster end to the target cluster end, generates target table metadata according to the metadata DDL information, and generates initial table data according to the target table metadata.

[0191] In an optional embodiment of this application, the copy mapping relationship determination module 505 is specifically used for:

[0192] Based on the source database system table, determine the Internet Protocol (IP) information for each node in the source cluster. Based on the target database system table, determine the IP information for each node in the target cluster. Based on the IP information of each node in the source cluster and each node in the target cluster, determine the replication mapping relationship between each node in the source cluster and each node in the target cluster.

[0193] In an optional embodiment of this application, the request response module 501, in response to the data copy request, is further configured to: generate statistical information of the source data.

[0194] In an optional embodiment of this application, the statistical information of the source data includes the number of records in the source table. Accordingly, the first sending module 507 is further specifically configured to: receive the target record count sent by the target cluster, where the target record count is generated when the target cluster imports the source table data into the initial table data; perform a data comparison process between the target record count and the source table record count to generate a record count comparison result; if the record count comparison result is determined to meet a preset error condition, then return to the step of generating replication configuration parameters and node activation instructions in response to the data replication request; if the record count comparison result is determined to meet a preset success condition, then return a data replication success result.

[0195] In an optional embodiment of this application, the statistical information of the source data includes the number of records in the source table. Accordingly, the first sending module 507 is further configured to: if it is determined that the number of records in the source table meets the preset threshold condition, then according to the replication mapping relationship, send the source table data from one of the nodes of the source cluster to the target cluster through the data transmission channel, so that the target cluster can import the source table data into the initial table data according to the target table metadata to generate the target table data.

[0196] In an optional embodiment of this application, the first sending module 507 is further configured to: send a same-table query instruction to the target cluster end, so that the target cluster end can determine the target end data table with the same source table metadata according to the same-table query instruction, and delete the target end data table.

[0197] In an optional embodiment of this application, the data splitting module 506 is further configured to: if it is determined that the number of nodes in the source cluster is not equal to the number of nodes in the target cluster, then redistribute the source table data to generate redistributed source table data.

[0198] In an optional embodiment of this application, the first sending module 507 is further configured to: send a statistical information collection instruction to the target cluster terminal, so that the target cluster terminal collects statistical information of the target table data according to the statistical information collection instruction.

[0199] refer to Figure 6 , Figure 6 This is a schematic diagram of another data copying device provided in an embodiment of this application. Figure 6 As shown, the data copying device 60 includes: a second receiving module 601, a second transmission channel establishment module 602, a second sending module 603, and a data copying module 604.

[0200] The second receiving module 601 is used to receive the node activation instruction sent by the source cluster end. The node activation instruction is generated by the source cluster end in response to the data replication request, and the replication configuration parameters are used to instruct the source cluster end to determine the source table data of the source data according to the replication configuration parameter information, and to determine the source table metadata according to the source database system table.

[0201] The second transmission channel establishment module 602 is used to establish a data transmission channel between each node of the source cluster and each node of the target cluster according to the node activation command.

[0202] The second sending module 603 is used to send the target database system table to the source cluster end, so that the source cluster end can determine the replication mapping relationship between the nodes of the source cluster end and the nodes of the target cluster end based on the target database system table and the source database system table, and split the source table data according to the replication mapping relationship.

[0203] The second receiving module 601 is also used to receive source table metadata sent by the source cluster end, generate target table metadata based on the source table metadata, and generate initial table data based on the target table metadata.

[0204] The second receiving module 601 is also used to receive the split source table data sent by the source cluster end through the data transmission channel, wherein the split source table data is sent from multiple nodes of the source cluster end to multiple nodes of the target cluster end.

[0205] The data replication module 604 is used to import source table data into initial table data based on target table metadata to generate target table data.

[0206] In an optional embodiment of this application, the second transmission channel establishment module 602 is specifically configured to: receive a node activation instruction sent by the source cluster end, wherein the node activation instruction is used by the source cluster end to activate each node of the source cluster end and determine the database super user whitelist information; activate each node corresponding to all nodes of the source cluster end according to the node activation instruction; receive the database super user whitelist information sent by the source cluster end, and determine the network access correspondence based on the database super user whitelist information; and send the network access correspondence to the source cluster end, so that the source cluster end establishes a data transmission channel between each node of the source cluster end and each node of the target cluster end according to the network access correspondence.

[0207] In an optional embodiment of this application, the source table metadata includes metadata database schema definition language (DDL) information. Accordingly, the second receiving module 601 is specifically configured to: receive the metadata DDL information sent from the source cluster, and execute a pre-stored synchronization tool based on the database superuser whitelist information to synchronize the source table metadata from the source cluster to the target cluster. Based on the metadata DDL information, target table metadata is generated, and based on the target table metadata, initial table data is generated.

[0208] In an optional embodiment of this application, the source cluster end, in response to the data replication request, also generates statistical information of the source data, including the number of records in the source table. Correspondingly, the data replication module 604 is further configured to: generate a target number of records when importing source table data into initial table data; send the target number of records to the source cluster end, so that the source cluster end performs data comparison processing between the target number of records and the number of records in the source table, generates a record number comparison result; and if the record number comparison result is determined to meet a preset error condition, return to the step of generating replication configuration parameters and node activation instructions in response to the data replication request; and if the record number comparison result is determined to meet a preset success condition, return a data replication success result.

[0209] In an optional embodiment of this application, the second receiving module 601 is further configured to: receive source table data sent by one of the nodes of the source cluster through a data transmission channel when the source cluster determines that the number of records in the source table meets a preset threshold condition. The data replication module 604 is configured to: import the source table data into the initial table data according to the target table metadata to generate the target table data.

[0210] In an optional embodiment of this application, the second receiving module 601 is further configured to: receive a same-table query instruction sent by the source cluster end; determine the target end data table with the same source table metadata according to the same-table query instruction; and delete the target end data table.

[0211] In an optional embodiment of this application, the second receiving module 601 is further configured to: receive a statistical information instruction sent by the source cluster end, and collect statistical information of the target table data according to the statistical information instruction.

[0212] The data copying device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0213] refer to Figure 7 , Figure 7 This is a schematic diagram of the hardware structure of the server system provided in the embodiments of this application, such as... Figure 7 As shown, the server system includes at least one processor 701 and a memory 702.

[0214] Among them, memory 702 is used to store computer execution instructions.

[0215] The processor 701 is configured to execute computer execution instructions stored in the memory 702 to implement the various steps involved in the above method embodiments. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0216] Alternatively, the memory 702 can be either standalone or integrated with the processor 701.

[0217] When the memory 702 is set up independently, the server system also includes a bus 703 for connecting the memory 702 and the processor 701.

[0218] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data copying method described above.

[0219] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the data copying method described above.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or modules, and may be electrical, mechanical, or other forms.

[0221] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0222] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0223] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods of the various embodiments of this application.

[0224] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0225] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0226] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0227] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0228] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0229] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0230] This description is intended to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data copying method, characterized in that, Applied to the source cluster end, including: In response to a data replication request, generate replication configuration parameters and node activation instructions; Based on the replication configuration parameter information, determine the source table data of the source data; based on the source database system table, determine the source table metadata; the source database system table is a table automatically generated by the system when storing data on the source cluster end, used to store system information and the corresponding data storage location information in the database; According to the node activation command, a data transmission channel is established between each node of the source cluster and each node of the target cluster. Receive the target database system table sent by the target cluster terminal; the target database system table is a table automatically generated by the system when the target cluster terminal stores data, used to store system information and the corresponding data storage location information in the database; Based on the source database system table, determine the Internet protocol information of each node in the source cluster. Based on the target database system table, determine the Internet protocol information of each node in the target cluster. Based on the Internet Protocol information of each node in the source cluster and the Internet Protocol information of each node in the target cluster, a replication mapping relationship between each node in the source cluster and each node in the target cluster is determined; the replication mapping relationship is used to represent the correspondence between a node in the source cluster and a node in the target cluster. Based on the replication mapping relationship, the source table data is split; The source table metadata is sent to the target cluster, so that the target cluster generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata. The split source table data is sent from multiple nodes in the source cluster to multiple nodes in the target cluster through the data transmission channel, so that the target cluster can import the source table data into the initial table data according to the target table metadata to generate the target table data.

2. The method according to claim 1, characterized in that, The step of establishing a data transmission channel between each node in the source cluster and each node in the target cluster according to the node activation command includes: According to the node activation instruction, each node on the source cluster is activated; the node activation instruction is sent to the target cluster so that the target cluster activates each node corresponding to all nodes on the source cluster according to the node activation instruction; the database super user whitelist information is determined according to the node activation instruction. The database super user whitelist information is sent to the target cluster terminal so that the target cluster terminal can determine the network access correspondence based on the database super user whitelist information. The system receives the network access correspondence sent by the target cluster and establishes a data transmission channel between each node of the source cluster and each node of the target cluster based on the network access correspondence.

3. The method according to claim 2, characterized in that, The source table metadata includes metadata database schema definition language (DDL) information; Accordingly, sending the source table metadata to the target cluster so that the target cluster generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata, includes: The metadata DDL information is sent to the target cluster, so that the target cluster executes the pre-stored synchronization execution tool according to the database super user whitelist information, synchronizes the source table metadata from the source cluster to the target cluster, generates target table metadata according to the metadata DDL information, and generates initial table data according to the target table metadata.

4. The method according to claim 1, characterized in that, The response to the data replication request also includes: Generate statistical information about the source data.

5. The method according to claim 4, characterized in that, The statistical information of the source data includes the number of records in the source table; Accordingly, after the split source table data is sent from multiple nodes in the source cluster to multiple nodes in the target cluster via the data transmission channel, so that the target cluster imports the source table data into the initial table data according to the target table metadata and generates the target table data, the process further includes: Receive the target record number sent by the target cluster terminal, wherein the target record number is generated when the target cluster terminal imports source table data into initial table data; The target number of records and the number of records in the source table are compared to generate a record count comparison result. If the record count comparison result is determined to meet the preset error reporting conditions, then return to the step of responding to the data replication request and generating replication configuration parameters and node activation instructions; If the record count comparison result is determined to meet the preset success conditions, a data copy success result is returned.

6. The method according to claim 4, characterized in that, The statistical information of the source data includes the number of records in the source table; Accordingly, after sending the source table metadata to the target cluster so that the target cluster generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata, the process further includes: If the number of records in the source table meets the preset threshold condition, then according to the replication mapping relationship, the source table data is sent from one of the nodes in the source cluster to the target cluster through the data transmission channel, so that the target cluster imports the source table data into the initial table data according to the target table metadata and generates the target table data.

7. The method according to claim 1, characterized in that, After sending the source table metadata to the target cluster so that the target cluster generates target table metadata based on the source table metadata and generates initial table data based on the target table metadata, the process further includes: A same-table query instruction is sent to the target cluster, so that the target cluster can determine the target data table with the same metadata as the source table based on the same-table query instruction, and delete the target data table.

8. The method according to claim 1, characterized in that, Before splitting the source table data according to the replication mapping relationship, the process further includes: If it is determined that the number of nodes in the source cluster is not equal to the number of nodes in the target cluster, the source table data is redistributed to generate redistributed source table data.

9. The method according to any one of claims 1 to 8, characterized in that, The step of sending the split source table data from multiple nodes in the source cluster to multiple nodes in the target cluster via the data transmission channel, so that the target cluster imports the source table data into the initial table data according to the target table metadata and generates the target table data, further includes: A statistical information collection instruction is sent to the target cluster terminal so that the target cluster terminal collects statistical information of the target table data according to the statistical information collection instruction.

10. A data copying method, characterized in that, Applied to the target cluster, including: The system receives a node activation instruction sent by the source cluster, wherein the node activation instruction is generated by the source cluster in response to a data replication request, and the replication configuration parameters are used to instruct the source cluster to determine the source table data of the source data based on the replication configuration parameter information; and to determine the source table metadata based on the source database system table. According to the node activation command, a data transmission channel is established between each node of the source cluster and each node of the target cluster. The target database system table is sent to the source cluster end, so that the source cluster end can determine the Internet Protocol (IP) information of each node of the source cluster end based on the source database system table, determine the IP information of each node of the target cluster end based on the target database system table, and determine the replication mapping relationship between each node of the source cluster end and each node of the target cluster end based on the IP information of each node of the source cluster end and the IP information of each node of the target cluster end. The source table data is then split according to the replication mapping relationship. The source database system table is a table automatically generated by the system when storing data on the source cluster end, used to store system information and the corresponding data storage location information in the database. The target database system table is a table automatically generated by the system when storing data on the target cluster end, used to store system information and the corresponding data storage location information in the database. The replication mapping relationship is used to represent the correspondence between a node on the source cluster end and a node on the target cluster end. Receive source table metadata sent by the source cluster end, generate target table metadata based on the source table metadata, and generate initial table data based on the target table metadata; Receive the split source table data sent by the source cluster end through the data transmission channel, wherein the split source table data is sent from multiple nodes of the source cluster end to multiple nodes of the target cluster end; Based on the target table metadata, the source table data is imported into the initial table data to generate the target table data.

11. The method according to claim 10, characterized in that, The step of establishing a data transmission channel between each node in the source cluster and each node in the target cluster according to the node activation command includes: Receive a node activation instruction sent by the source cluster terminal, wherein the node activation instruction is used by the source cluster terminal to enable each node of the source cluster terminal and determine the database super user whitelist information; According to the node activation command, each node corresponding to all nodes in the source cluster is activated; Receive the database super user whitelist information sent by the source cluster end, and determine the network access correspondence based on the database super user whitelist information; The network access relationship is sent to the source cluster end, so that the source cluster end can establish a data transmission channel between each node of the source cluster end and each node of the target cluster end according to the network access correspondence relationship.

12. The method according to claim 11, characterized in that, The source table metadata includes metadata database schema definition language (DDL) information; Accordingly, the step of receiving source table metadata sent by the source cluster end, generating target table metadata based on the source table metadata, and generating initial table data based on the target table metadata includes: The system receives metadata DDL information sent from the source cluster and executes a pre-stored synchronization tool based on the database super user whitelist information to synchronize the source table metadata from the source cluster to the target cluster. Based on the metadata DDL information, generate target table metadata, and based on the target table metadata, generate initial table data.

13. The method according to claim 10, characterized in that, In response to the data replication request, the source cluster also generates statistical information about the source data, which includes the number of records in the source table. Accordingly, after importing the source table data into the initial table data based on the target table metadata to generate the target table data, the process further includes: When importing data from the source table into the initial table, the target number of records is generated; The target number of records is sent to the source cluster, so that the source cluster performs data comparison processing on the target number of records and the number of records in the source table, generates a record number comparison result, and if the record number comparison result is determined to meet the preset error condition, it returns to the step of responding to the data replication request, generating replication configuration parameters and node activation instructions, and if the record number comparison result is determined to meet the preset success condition, it returns a data replication success result.

14. The method according to claim 13, characterized in that, After importing source table data into initial table data based on the target table metadata to generate target table data, the process further includes: When the source cluster determines that the number of records in the source table meets the preset threshold condition, it receives the source table data sent by one of the nodes of the source cluster through the data transmission channel. Based on the target table metadata, import the source table data into the initial table data to generate the target table data.

15. The method according to claim 10, characterized in that, After generating target table metadata based on the source table metadata, and generating initial table data based on the target table metadata, the process further includes: Receive the same table query instruction sent by the source cluster terminal; Based on the same table query instruction, the target data table with the same metadata as the source table is determined, and the target data table is deleted.

16. The method according to any one of claims 10 to 15, characterized in that, After importing source table data into initial table data based on the target table metadata to generate target table data, the process further includes: Receive statistical information instructions sent by the source cluster end, and collect statistical information of the target table data according to the statistical information instructions.

17. A data copying device, characterized in that, Applied to the source cluster end, including: The request and response module is used to respond to data replication requests and generate replication configuration parameters and node activation instructions; The source data processing module is used to determine the source table data of the source data according to the replication configuration parameter information; and to determine the source table metadata according to the source database system table; the source database system table is a table automatically generated by the system when storing data on the source cluster end, which is used to store system information and the corresponding data storage location information in the database; The first transmission channel establishment module is used to establish a data transmission channel between each node of the source cluster and each node of the target cluster according to the node activation instruction. The first receiving module is used to receive the target database system table sent by the target cluster terminal; the target database system table is a table automatically generated by the system when the target cluster terminal stores data, used to store system information and the corresponding data storage location information in the database; The replication mapping relationship determination module is used to determine the Internet Protocol (IP) information of each node in the source cluster based on the source database system table; determine the IP information of each node in the target cluster based on the target database system table; and determine the replication mapping relationship between each node in the source cluster and each node in the target cluster based on the IP information of each node in the source cluster and the IP information of each node in the target cluster. The replication mapping relationship is used to represent the correspondence between a node in the source cluster and a node in the target cluster. The data splitting module is used to split the source table data according to the replication mapping relationship; The first sending module is used to send the source table metadata to the target cluster end, so that the target cluster end generates target table metadata based on the source table metadata, and generates initial table data based on the target table metadata; The first sending module is further configured to send the split source table data from multiple nodes of the source cluster to multiple nodes of the target cluster through the data transmission channel, so that the target cluster can import the source table data into the initial table data and generate the target table data according to the target table metadata.

18. A data copying device, characterized in that, Applied to the target cluster, including: The second receiving module is used to receive a node activation instruction sent by the source cluster end, wherein the node activation instruction is generated by the source cluster end in response to the data replication request, and the replication configuration parameters are used to instruct the source cluster end to determine the source table data of the source data according to the replication configuration parameter information; and to determine the source table metadata according to the source database system table. The second transmission channel establishment module is used to establish a data transmission channel between each node of the source cluster and each node of the target cluster according to the node activation instruction. The second sending module is used to send the target database system table to the source cluster end, so that the source cluster end can determine the Internet Protocol information of each node of the source cluster end according to the source database system table, determine the Internet Protocol information of each node of the target cluster end according to the target database system table, and determine the replication mapping relationship between each node of the source cluster end and each node of the target cluster end according to the Internet Protocol information of each node of the source cluster end and the Internet Protocol information of each node of the target cluster end; and split the source table data according to the replication mapping relationship. The source database system table is a table automatically generated by the system when storing data on the source cluster end, used to store system information and the corresponding data storage location information in the database. The target database system table is a table automatically generated by the system when storing data on the target cluster end, used to store system information and the corresponding data storage location information in the database. The replication mapping relationship is used to represent the correspondence between a node on the source cluster end and a node on the target cluster end. The second receiving module is further configured to receive source table metadata sent by the source cluster end, generate target table metadata based on the source table metadata, and generate initial table data based on the target table metadata; The second receiving module is further configured to receive the split source table data sent by the source cluster end through the data transmission channel, wherein the split source table data is sent from multiple nodes of the source cluster end to multiple nodes of the target cluster end; The data replication module is used to import source table data into initial table data based on the target table metadata, thereby generating target table data.

19. A server system, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data copying method as described in any one of claims 1 to 9 or any one of claims 10 to 16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data copying method as described in any one of claims 1 to 9 or any one of claims 10 to 16.

21. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data copying method as described in any one of claims 1 to 9 or any one of claims 10 to 16.

Images