Data replication method, apparatus, device, and computer storage medium

Abstract

The application discloses a data replication method and device, equipment and a computer storage medium. The method comprises the following steps: obtaining configuration information of a data table to be replicated, wherein the configuration information at least comprises a source cluster address, a target cluster address and a data table structure of the data table to be replicated; querying the data table structure of the data table to be replicated, and determining partition information of the data table to be replicated; partitioning the data table to be replicated based on the partition information, and obtaining a plurality of data areas; and taking the data area as a replication unit, and replicating data in the data table to be replicated from a source cluster corresponding to the source cluster address to a target cluster corresponding to the target cluster address. According to the embodiment of the application, the data table is replicated by taking the data area as the replication unit, the time for data replication query can be saved, and therefore the replication efficiency of the data table to be replicated is improved under the same environmental resource condition.

Description

Technical Field

[0001] This application belongs to the field of data storage, and in particular relates to a data copying method, apparatus, device, and computer storage medium. Background Technology

[0002] With the advent of the Internet era, the upgrade of database hardware and software could not meet the explosive growth of business needs and data volume. As a result, big data platforms have emerged. In increasingly complex business scenarios and disaster recovery construction scenarios, the need for large-scale data migration and replication between clusters is inevitable.

[0003] Currently, stored procedures in Massively Parallel Processing (MPP) architectures can support data replication between clusters. However, most existing replication methods replicate data by using the entire data table as the replication unit, which results in low data table replication efficiency when environmental resources are limited. Summary of the Invention

[0004] This application provides a data copying method, apparatus, device, and computer storage medium. By using data areas as copying units to copy data tables, the time for data copying queries can be saved, thereby improving the data table copying efficiency under the same environmental resource conditions.

[0005] In a first aspect, embodiments of this application provide a data copying method, the method comprising:

[0006] Obtain the configuration information of the data table to be copied. The configuration information includes at least the source cluster address, target cluster address, and data table structure of the data table to be copied.

[0007] Query the data table structure of the data table to be copied, and determine the partition information of the data table to be copied;

[0008] The data table to be copied is partitioned based on the partition information to obtain multiple data areas;

[0009] Using the data area as the replication unit, the data in the data table to be replicated is copied from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address.

[0010] In some possible implementations of the first aspect, the partitioning information includes at least data range pruning conditions and data intervals, querying the data table structure of the data table to be copied, and determining the partitioning information of the data table to be copied, including:

[0011] Query the data table structure of the table to be copied to determine the partition key information;

[0012] Based on the partition key information and the business replication requirements of the data table to be replicated, determine the data area pruning conditions and data range of the data table to be replicated.

[0013] In some possible implementations of the first aspect, the data table to be copied is partitioned based on partitioning information to obtain multiple data areas, including:

[0014] When using the invariant operator in the data area pruning conditions corresponding to each data area, the data table to be copied is partitioned based on the data area pruning conditions and the data area, resulting in multiple data areas.

[0015] In some possible implementations of the first aspect, after partitioning the data table to be copied based on partitioning information to obtain multiple data areas, the method further includes:

[0016] The script calls a preset condition file based on the copy script, and stores the data area trimming conditions into the preset condition file according to the format of the preset condition file.

[0017] Based on the format of the preset condition file, the data areas in the data table to be copied are sorted out according to the data area clipping conditions, and the data to be copied in each data area is determined.

[0018] In some possible implementations of the first aspect, data in the data table to be copied is copied from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address, using the data area as the copy unit. This includes:

[0019] Based on the data table structure of the data area, create a temporary table and a target table in the target cluster that have the same data table structure as the data area;

[0020] Read data from the data range corresponding to each data area in the data table to be copied, and copy the data to be copied to the temporary table corresponding to each data area;

[0021] Once replication is complete in each data area, the data to be replicated from the temporary table corresponding to each data area is inserted into the target table corresponding to the target cluster address, and then the temporary table is deleted.

[0022] In some possible implementations of the first aspect, if a data table with the same name as the data area exists in the target cluster, the method further includes:

[0023] Rename or delete data tables in the target cluster that have the same name as the table in the data area.

[0024] Secondly, embodiments of this application provide a data copying apparatus, the apparatus comprising:

[0025] The acquisition module is used to obtain the configuration information of the data table to be copied. The configuration information includes at least the source cluster address, target cluster address and data table structure of the data table to be copied.

[0026] The query module is used to query the data table structure of the data table to be copied and determine the partition information of the data table to be copied.

[0027] The partitioning module is used to partition the data table to be copied based on partitioning information, resulting in multiple data areas.

[0028] The replication module is used to copy data from the source cluster (located at the source cluster address) to the target cluster (located at the target cluster address), using the data area as the replication unit.

[0029] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions;

[0030] When the processor executes computer program instructions, it implements the data copying method as described in any one of the first aspects.

[0031] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the data copying method as described in any one of the first aspects.

[0032] Fifthly, embodiments of this application provide a computer program product in which the instructions are executed by a processor of an electronic device, enabling the electronic device to perform the data copying method described in any one of the first aspects.

[0033] The data replication method, apparatus, device, and computer storage medium of this application embodiment obtain partition information by querying the data table structure of the data table to be replicated. Based on this partition information, the data table to be replicated is partitioned into multiple data areas. Thus, data in the data table to be replicated is copied from the source cluster to the target cluster using data areas as the replication unit, rather than the entire data table to be replicated. This achieves data replication between clusters. Furthermore, because data areas have the characteristic of targeted querying rather than full table querying, the time for data replication querying can be saved during the data table replication process using data areas as replication units, thereby improving the replication efficiency of the data table to be replicated under the same environmental resource conditions. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a data copying method provided in an embodiment of this application;

[0036] Figure 2 This is a flowchart illustrating the specific implementation method of step S102;

[0037] Figure 3 This is a flowchart illustrating a method for obtaining a target format file provided in an embodiment of this application;

[0038] Figure 4 This is a flowchart illustrating the specific implementation method of step S104;

[0039] Figure 5 This is a schematic diagram of the structure of a data copying device provided in an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0042] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0043] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0044] With the advent of the internet age, the upgrades to database hardware and software could no longer meet the explosive growth in business demands and data volume. As a result, big data platforms emerged, and data middleware and MPP databases gradually gained popularity. Longfu MPP database is a distributed database based on the MPP architecture, suitable for typical data warehouse scenarios, and adept at complex analytical queries under large data volumes. The MPP database can be divided into two modules: a management module (the MPP management console) and a user module (the tenant cluster, including metadata clusters and computing clusters).

[0045] In increasingly complex business scenarios and disaster recovery construction scenarios, there is an inevitable need for large-scale data migration and replication between clusters. The existing stored procedure copy_table_between_clusters in MPP databases can support data replication between clusters, but under limited environmental resources, the efficiency and success rate of data table replication are low.

[0046] To address the problems of the prior art, embodiments of this application provide a data copying method, apparatus, device, and computer storage medium. The data copying method provided in this application embodiment will be described first below.

[0047] Figure 1 A flowchart illustrating a data copying method provided in an embodiment of this application is shown. Figure 1 As shown, the method specifically includes the following steps:

[0048] S101. Obtain the configuration information of the data table to be copied. The configuration information shall include at least the source cluster address, target cluster address and data table structure of the data table to be copied.

[0049] S102. Query the data table structure of the data table to be copied, and determine the partition information of the data table to be copied;

[0050] S103. Based on the partitioning information, partition the data table to be copied to obtain multiple data areas;

[0051] S104. Using the data area as the replication unit, copy the data in the data table to be replicated from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address.

[0052] Therefore, according to the data replication method provided in this application, partition information is obtained by querying the data table structure of the data table to be replicated. Based on this partition information, the data table to be replicated is partitioned into multiple data areas. Thus, data in the data table to be replicated is copied from the source cluster to the target cluster using data areas as the replication unit, rather than the entire data table to be replicated. This achieves data replication between clusters. Furthermore, because data areas have the characteristic of targeted queries rather than full table queries, the time for data replication queries can be saved during the data table replication process using data areas as replication units, thereby improving the replication efficiency of the data table to be replicated under the same environmental resource conditions.

[0053] In some embodiments, in S101, before copying the data table to be copied, the source cluster IP address, target cluster IP address, and data table structure corresponding to the data table to be copied in the database must first be obtained. Here, the source cluster refers to the cluster where the data table to be copied is currently located, and the target cluster refers to the cluster to which the data table to be copied will be migrated. In addition to obtaining the above configuration information, to improve the efficiency and accuracy of the copying, auxiliary information such as database users and data tablespace information can also be obtained.

[0054] In some embodiments, in S102, the partitioning information includes at least data area pruning conditions and data ranges, which can be used to determine the data partitioning rules for the data area. As an example, such as... Figure 2 As shown, the above S102 may specifically include the following steps:

[0055] S1021. Query the data table structure of the data table to be copied and determine the partition key information;

[0056] S1022. Based on the partition key information and combined with the business replication requirements of the data table to be replicated, determine the data area pruning conditions and data range of the data table to be replicated.

[0057] As an example, in S1021 and S1022, the business replication requirement corresponds to the database user's requirement, that is, how much data needs to be replicated. The partition key information mainly targets the partition key field. Combined with the data table structure of the data table to be replicated, if the database user prefers to obtain data from the data table to be replicated within a certain period of time, then the corresponding data area pruning condition is related to the date. For example, the data area pruning condition can be dateFilter = "date > date '2021-01-01' and date < '2021-02-01'", then the corresponding partition key field is "date", and the data range is from 2021-01-01 to 2021-02-01.

[0058] Therefore, by querying the data table structure of the data table to be copied and determining the data area pruning conditions and data ranges in combination with business replication requirements, the data table to be copied is partitioned using these conditions, resulting in multiple data areas, which makes it easier to manage the data table to be copied.

[0059] In some embodiments, in S103, the partitioning operation is performed by the optimizer. When the invariant operator is used in the data area pruning condition corresponding to each partition table, the data table to be copied is partitioned using the data area pruning condition and the data area to obtain multiple data areas. As an example, using date as the partition key field, the data range is a calendar month, such as from 2021-01-01 to 2021-02-01, and the operators in the data range clipping conditions corresponding to each data range are the same. The operators can be =, <, <=, >=, <>, etc. For example, the data range clipping condition corresponding to the data range from 2021-01-01 to 2021-02-01 is dateFilter = "date > date '2021-01-01' and date < '2021-02-01'", then the data range clipping condition corresponding to the data range from 2021-02-01 to 2021-03-01 is dateFilter = "date > date '2021-02-01' and date < '2021-03-01'", and the operators involved in the two are the same, ">" and <.

[0060] Therefore, while ensuring the continuity of partition pruning, the optimizer can perform partition pruning. Partition pruning is an optimization method for partitioned tables; during data replication, only partitions that meet the conditions are scanned, thereby reducing the amount of data to be copied. This facilitates subsequent data replication based on data areas, effectively improving replication efficiency. Furthermore, replication based on data areas reduces the number of rollbacks. In the event of a replication failure, only the corresponding data area needs to be queried, without needing to roll back the entire table to be copied. This reduces the amount of data to be copied while also increasing the success rate of replication.

[0061] In some embodiments, to ensure that the optimizer can perform partition pruning, such as Figure 3 As shown, after S103, the method may further include the following steps:

[0062] S301. Based on the copy script, call the preset condition file and store the data area trimming conditions into the preset condition file according to the format of the preset condition file.

[0063] S302. Based on the format of the preset condition file, sort out the data areas in the data table to be copied according to the data area trimming conditions, and determine the data to be copied in each data area.

[0064] As an example, in S301 and S302, the preset condition file can be a list file, and the format of the preset condition file can be a list file. After determining the data area pruning conditions, the data area pruning conditions are stored in the data table to be copied, based on the data table structure. The list file format includes the table name, table name (tname), and data copying condition (dateFilter). Therefore, based on the list file format, the data areas in the data table to be copied, divided according to the data area pruning conditions, are sorted out to determine the data to be copied in each data area.

[0065] Therefore, the multiple data areas obtained after partitioning the data table to be copied are sorted according to a unified list file format so that the optimizer can perform partition pruning according to the list format file, ensuring the smooth progress of partition pruning, thereby reducing the number of failed rollbacks and the waste of resources and time.

[0066] In some embodiments, in S104, the data area is used as the copying unit to copy the data in the data table to be copied from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address, so as to realize data copying between clusters in the MPP database.

[0067] As an example, such as Figure 4 As shown, the above S104 may specifically include the following steps:

[0068] S1041. Based on the data table structure of the data area, create a temporary table and a target table in the target cluster that have the same data table structure as the partition table;

[0069] S1042. Read data from the data range corresponding to each data area in the data table to be copied, and copy the data to be copied to the temporary table corresponding to each data area;

[0070] S1043. After replication is completed in each data area, insert the data to be replicated from the temporary table corresponding to each data area into the target table corresponding to the target cluster address, and then delete the temporary table.

[0071] For example, in the process of data replication based on the stored procedure copy_table_between_cluster, the target table adopts a random distribution strategy, while the data area contains a specified distribution key. Therefore, a temporary table is used as an intermediary. At the same time, creating a temporary table can also increase concurrency, that is, multiple processes can be started at the same time to replicate the data table to be replicated, thereby improving the data replication efficiency.

[0072] In addition, to avoid replication failures due to identical table names, if a table with the same name as the data area exists in the target cluster during the data replication process, the table with the same name in the target cluster will be renamed or deleted.

[0073] Therefore, based on data area pruning conditions and data ranges, the data table to be copied is divided into many resource-efficient and easily managed data areas, each corresponding to an independent data table. When executing queries containing partition key fields, leveraging the characteristics of data areas, only the portion of data that meets the data area pruning conditions is scanned, rather than the entire table of data to be copied, thus improving query performance. Furthermore, utilizing these characteristics of data areas, when copying data using the stored procedure `copy_table_between_cluster`, data areas are used as the copy unit. Information statistics and organization are performed after the entire table is copied. Under the same environmental resources, this effectively improves the copying efficiency and success rate of the data table to be copied. Moreover, this process is simple to configure and operate, facilitating implementation and thus improving work efficiency.

[0074] It should be noted that the application scenarios described in the above embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0075] Based on the same inventive concept, this application provides a data copying device, specifically combined with... Figure 5 A detailed explanation will be provided.

[0076] Figure 5 This application illustrates a data copying apparatus provided in an embodiment, such as... Figure 5 As shown, the data copying device 500 may include:

[0077] The acquisition module 501 is used to acquire the configuration information of the data table to be copied. The configuration information includes at least the source cluster address, target cluster address and data table structure of the data table to be copied.

[0078] The query module 502 is used to query the data table structure of the data table to be copied and determine the partition information of the data table to be copied.

[0079] Partitioning module 503 is used to partition the data table to be copied based on partitioning information to obtain multiple data areas;

[0080] The replication module 504 is used to replicate data in the data table to be replicated from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address, using the data area as the replication unit.

[0081] Therefore, the data replication apparatus provided in this application obtains partition information by querying the data table structure of the data table to be replicated, and then partitions the data table to be replicated based on the partition information to obtain multiple data areas. Thus, data in the data table to be replicated is copied from the source cluster to the target cluster using data areas as the replication unit, rather than the entire data table to be replicated, thereby achieving data replication between clusters. Furthermore, because data areas have the characteristic of targeted querying rather than full table querying, the time for data replication querying can be saved during the data table replication process using data areas as replication units, thereby improving the data table replication efficiency under the same environmental resource conditions.

[0082] In some embodiments, the partitioning information includes at least data area pruning conditions and data ranges, and the query module 502 may include the following sub-modules:

[0083] The first determination submodule is used to query the data table structure of the data table to be copied and determine the partition key information;

[0084] The second determining submodule is used to determine the data area pruning conditions and data range of the data table to be copied based on the partition key information and in combination with the business replication requirements corresponding to the data table to be copied.

[0085] In some embodiments, in order to partition the data table to be copied to obtain multiple data areas, the partitioning module 503 may include the following sub-modules:

[0086] The partitioning submodule is used to partition the data table to be copied based on the data area pruning conditions and the data area range, when using the invariant operator in the data area pruning conditions corresponding to the data area of ​​each partition table, to obtain multiple data areas.

[0087] In some embodiments, in order to determine the data to be copied in each data area, the partitioning module 503 may further include the following sub-modules:

[0088] The calling submodule is used to call the preset condition file based on the copy script, and store the data area clipping conditions into the preset condition file according to the format of the preset condition file;

[0089] The sorting submodule is used to sort out the data areas in the data table to be copied based on the data area trimming conditions according to the format of the preset condition file, and determine the data to be copied in each data area.

[0090] In some embodiments, the copying module 504 may include the following sub-modules:

[0091] Create a submodule to create a temporary table and a target table in the target cluster that have the same data table structure as the data area, based on the data table structure of the data area;

[0092] The read submodule is used to read data from the data range corresponding to each data area in the data table to be copied, and copy the data to be copied to the temporary table corresponding to each data area;

[0093] The processing submodule is used to insert the data to be copied from the temporary table corresponding to each data area into the target table corresponding to the target cluster address, and delete the temporary table after the replication of each data area has been completed.

[0094] In some embodiments, if a data table with the same name as the data area exists in the target cluster, the above processing submodule may further include the following units:

[0095] The processing unit is used to rename or delete data tables in the target cluster that have the same table name as the data area.

[0096] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown.

[0097] The electronic device 600 may include a processor 601 and a memory 602 storing computer program instructions.

[0098] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0099] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0100] Memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 602 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the data copying method according to the first aspect of this application.

[0101] The processor 601 implements any of the data copying methods described in the above embodiments by reading and executing computer program instructions stored in the memory 602.

[0102] In one example, electronic device 600 may further include communication interface 603 and bus 604. Wherein, as... Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 604 and complete communication with each other.

[0103] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0104] Bus 604 includes hardware, software, or both, that couples components of electronic device 600 together. For example, and not as a limitation, bus 604 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0105] This electronic device 600 can achieve integration Figure 1 and Figure 5 The described data copying method and apparatus.

[0106] Furthermore, in conjunction with the data copying methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data copying methods in the above embodiments.

[0107] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0108] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0109] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0110] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0111] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data copying method, characterized in that, include: Obtain the configuration information of the data table to be copied, wherein the configuration information includes at least the source cluster address, target cluster address, and data table structure of the data table to be copied; Query the data table structure of the data table to be copied to determine the partition information of the data table to be copied; The data table to be copied is partitioned based on the partitioning information to obtain multiple data areas; Using the data area as a copying unit, the data in the data table to be copied is copied from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address; The partitioning information includes at least the data area pruning conditions and the data range. The step of querying the data table structure of the data table to be copied and determining the partition information of the data table to be copied includes: Query the data table structure of the data table to be copied to determine the partition key information, which includes a distinguishing key field. Based on the partition key information and in conjunction with the business replication requirements corresponding to the data table to be replicated, the data area pruning conditions and data range of the data table to be replicated are determined, and the business replication requirements correspond to the database user's requirements. Based on the distinguishing key field and the data range, a data range pruning condition is generated for each data range, so that the optimizer performs a partitioning operation based on the data range pruning condition to obtain multiple data ranges, wherein the operators corresponding to the multiple data range pruning conditions are the same; The script calls a preset condition file based on the copy script, and stores the data area clipping conditions into the preset condition file according to the format of the preset condition file. Based on the format of the preset condition file, the data areas in the data table to be copied are sorted out according to the data area pruning conditions, and the data to be copied in each data area is determined. The preset condition file is a list file, and the format of the preset condition file is a list file format; After determining the data area pruning conditions, the data area pruning conditions are stored in the data table to be copied, based on the data table structure of the data table to be copied.

2. The method according to claim 1, characterized in that, The data table to be copied is partitioned based on the partition information to obtain multiple data areas, including: When using an invariant operator in the data area pruning condition corresponding to each data area, the data table to be copied is partitioned based on the data area pruning condition and the data range to obtain multiple data areas.

3. The method according to claim 2, characterized in that, The step of using the data area as a replication unit to replicate data in the data table to be replicated from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address includes: Based on the data table structure of the data area, a temporary table and a target table with the same data table structure as the data area are created in the target cluster; Read data from the data interval corresponding to each data area in the data table to be copied, and copy the data to be copied to the temporary table corresponding to each data area; Once replication is complete in each of the data areas, the data to be replicated from the temporary table corresponding to each data area is inserted into the target table corresponding to the target cluster address, and the temporary table is deleted.

4. The method according to claim 3, characterized in that, If a table with the same name as the table in the data area exists in the target cluster, the method further includes: Rename or delete the data tables in the target cluster that have the same table name as the data area.

5. A data copying device, characterized in that, include: The acquisition module is used to acquire the configuration information of the data table to be copied. The configuration information includes at least the source cluster address, target cluster address, and data table structure of the data table to be copied. The query module is used to query the data table structure of the data table to be copied and determine the partition information of the data table to be copied. The partitioning module is used to partition the data table to be copied based on the partitioning information to obtain multiple data areas. The partitioning information includes at least data area pruning conditions and data ranges. It is also used to query the data table structure of the data table to be copied, determine the partition key information, the partition key information including the distinguishing key field; based on the partition key information, and in combination with the business copying requirements corresponding to the data table to be copied, determine the data area pruning conditions and data range of the data table to be copied; Based on the distinguishing key field and the data range, data range pruning conditions are generated for each data range, so that the optimizer performs partitioning operations based on the data range pruning conditions to obtain multiple data ranges, wherein the operators corresponding to the multiple data range pruning conditions are the same; the business replication requirement corresponds to the database user's requirement; it is also used to call a preset condition file based on the replication script, and store the data range pruning conditions in the preset condition file according to the format of the preset condition file; based on the format of the preset condition file, the data ranges in the data table to be replicated are sorted according to the data range pruning conditions, and the data to be replicated in each data range is determined; the preset condition file is a list file, and the format of the preset condition file is a list file format; it is also used to store the data range pruning conditions in the data table to be replicated after determining the data range pruning conditions, in combination with the data table structure of the data table to be replicated; The replication module is used to copy data from the source cluster corresponding to the source cluster address to the target cluster corresponding to the target cluster address, using the data area as the replication unit.

6. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data copying method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the data copying method as described in any one of claims 1-4.

8. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the data copying method as described in any one of claims 1-4.

Images