Data cross-database migration method and device

Abstract

The embodiment of the application discloses a data cross-database migration method and device, relates to the technical field of big data, and comprises the following steps: when a cross-database migration instruction for migrating target data in a first database to a second database is received, a currently idle process group is determined, wherein each process group comprises a first process and a second process, the first process is used for exporting data in the first database, and the second process is used for importing data into the second database; a first process in the currently idle process group is called to export the target data in the first database and store the target data into a preset transfer module; and a second process in the currently idle process group is called to import the target data stored in the transfer module into the second database. The application has the beneficial effect of simply and efficiently migrating data across databases.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and more specifically, to a method and apparatus for cross-database data migration. Background Technology

[0002] With technological advancements, big data platforms increasingly employ various database clusters to store diverse data types. For instance, relatively inexpensive Hadoop clusters are used to store source data, while MPP (Massively Parallel Processing) database clusters are used for storing topic-based data. As business scenarios become more diverse, scenarios frequently arise where data from multiple homogeneous or heterogeneous database clusters need to be correlated. For example, associating source data in a Hadoop cluster with topic data in an MPP database (MPPDB) cluster involves cross-cluster data access. Currently, this requires applications to use different tools to migrate data between databases. For instance, transferring data from a Hadoop cluster to an MPPDB cluster typically uses SQL on Hadoop technology for data transfer and loading; transferring data from an MPPDB cluster to a Hadoop cluster requires Sqoop; and transferring data from one MPPDB cluster to another requires GDS configuration for importing and exporting file data. The different data access directions for each cluster type necessitate different tools, resulting in complex operations, low efficiency, and high technical difficulty. Existing technologies lack a simple and efficient solution for cross-database data migration. Summary of the Invention

[0003] In order to solve at least one of the technical problems in the background art, the present invention proposes a method and apparatus for cross-database migration.

[0004] To achieve the above objectives, according to one aspect of the present invention, a method for cross-database migration is provided, the method comprising:

[0005] Upon receiving a cross-database migration instruction to migrate target data from a first database to a second database, the currently idle process groups are determined, wherein each process group includes: a first process and a second process, the first process being used to export data from the first database and the second process being used to import data into the second database;

[0006] The first process in the currently idle process group is invoked to export the target data from the first database and store it in a preset transfer module;

[0007] The second process in the currently idle process group is invoked to import the target data stored in the transfer module into the second database.

[0008] To achieve the above objectives, according to another aspect of the present invention, a data cross-database migration apparatus is provided, the apparatus comprising:

[0009] The process group determination unit is used to determine the currently idle process groups when receiving a cross-database migration instruction to migrate target data from a first database to a second database. Each process group includes a first process and a second process, wherein the first process is used to export data from the first database and the second process is used to import data into the second database.

[0010] The first processing unit is used to call the first process in the currently idle process group to export the target data in the first database and store it in a preset transfer module;

[0011] The second processing unit is used to call the second process in the currently idle process group to import the target data stored in the transfer module into the second database.

[0012] To achieve the above objectives, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data cross-database migration method.

[0013] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program / instructions are stored, which, when executed by a processor, implement the steps of the above-described data cross-database migration method.

[0014] To achieve the above objectives, according to another aspect of the present invention, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the above-described data cross-database migration method.

[0015] The beneficial effects of this invention are as follows:

[0016] In this embodiment of the invention, upon receiving a cross-database migration instruction, the first process in the currently idle process group is invoked to export the target data from the first database and store it in a preset transfer module. Then, the second process in the same currently idle process group is invoked to import the target data stored in the transfer module into the second database, thus achieving data migration. This invention eliminates the need for different tools, is simple to operate, and can automatically complete cross-database migration with high efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0018] Figure 1 This is a flowchart of the data cross-database migration method according to an embodiment of the present invention;

[0019] Figure 2 This is a flowchart illustrating how an embodiment of the present invention determines the currently idle process group;

[0020] Figure 3 This is a flowchart of process group locking and unlocking according to an embodiment of the present invention;

[0021] Figure 4 This is a flowchart illustrating the generation of folders in the transit module according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram illustrating the interaction of various components in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the process group in an embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram of cross-database migration according to an embodiment of the present invention;

[0025] Figure 8 This is a structural block diagram of the data cross-database migration device according to an embodiment of the present invention;

[0026] Figure 9 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0029] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0032] It should be noted that the data cross-database migration method and apparatus of the present invention can be used in the financial field, or in any field other than the financial field. The application field of the data cross-database migration method and apparatus of the present invention is not limited.

[0033] It should be noted that the following embodiments of the present invention use Hive database and MPP database as examples to illustrate how to achieve cross-database migration, but the present invention is not limited thereto.

[0034] Figure 1 This is a flowchart of the data cross-database migration method according to an embodiment of the present invention, as follows: Figure 1 As shown, in one embodiment of the present invention, the data cross-database migration method of the present invention includes steps S101 to S103.

[0035] Step S101: Upon receiving a cross-database migration instruction to migrate target data from the first database to the second database, determine the currently idle process groups, wherein each process group includes: a first process and a second process, the first process being used to export data from the first database, and the second process being used to import data into the second database.

[0036] In one embodiment of the present invention, the first process is further configured to import data into the first database, and the second process is further configured to export data from the second database.

[0037] In another embodiment of the present invention, the first database is an MPP database and the second database is a Hive database. In this case, the first process is the GDS process and the second process is the NODE process.

[0038] In one embodiment of the present invention, the first database is a Hive database and the second database is an MPP database. In this case, the first process is the NODE process and the second process is the GDS process.

[0039] In another embodiment of the present invention, both the first database and the second database are Hive databases, in which case both the first process and the second process are NODE processes.

[0040] In another embodiment of the present invention, both the first database and the second database are MPP databases, in which case both the first process and the second process are GDS processes.

[0041] Step S102: Invoke the first process in the currently idle process group to export the target data in the first database and store it in the preset transfer module.

[0042] In this invention, a relay module for data relay is preset, which is also referred to as a relay machine in this invention. This relay module can be a relay server.

[0043] In one embodiment of the present invention, the first process and the second process in each process group are both set in the relay module.

[0044] In one embodiment of the present invention, the first process and the second process correspond one-to-one in the relay machine, and their number is consistent with the number of logical disks in the relay machine, thereby achieving the goal of making full use of cluster resources.

[0045] Step S103: Invoke the second process in the currently idle process group to import the target data stored in the transfer module into the second database.

[0046] Therefore, this invention overcomes the shortcomings of existing cross-database data migration methods and provides a method that is simple to implement, low in cost, and can guarantee cross-database data migration.

[0047] Figure 5 This is a schematic diagram illustrating the interaction of various components in an embodiment of the present invention. The data cross-database migration method of the present invention is applied to... Figure 5 The control module, which is the execution entity of the cross-database migration method of this invention, is... Figure 5 The control module in the middle.

[0048] Figure 5 In the middle, the control module is responsible for access control and, based on the currently idle process group, sends the received data cross-database migration instructions to the corresponding relay module.

[0049] The first process is responsible for reading and writing to the first database, exporting data from the first database to a specified directory in the transit module, or loading files from a specified directory in the transit module into the first database.

[0050] The second process is responsible for reading and writing to the second database, exporting data from tables in the first database to a specified directory of the transfer module, or loading data from text files in a specified directory of the transfer module into tables in the specified second database.

[0051] like Figure 2 In one embodiment of the present invention, the determination of the currently idle process group in step S101 specifically includes steps S201 and S202.

[0052] Step S201: Obtain a preset first process control table, wherein each process group corresponds to a first data record in the first process control table, and the first data record includes a status flag bit. When the status flag bit is a first value, it indicates that the corresponding process group is in a locked state, and when the status flag bit is a second value, it indicates that the corresponding process group is in an idle state.

[0053] In one embodiment of the present invention, the first process control table may specifically be an Oracle table, and in some embodiments of the present invention, the first process control is referred to as Oracle table A.

[0054] In this invention, the first data record also contains a unique ID of the corresponding process group, which is represented by UNITID in this invention.

[0055] Step S202: Determine the currently idle process group based on the current value of the status flag bit in each data record of the first process control table.

[0056] In one embodiment of the present invention, each record in the first process control table corresponds to a unique UNIT ID value. Each record has a status flag bit, with the corresponding field name being STATUS. When the value of this field is 1, it indicates that the process group is locked. Simultaneously, the taskID (unique identifier corresponding to the cross-database migration instruction) using the process group is recorded in the LOCKER_ID field, and the locking time is recorded in the LOCKED_TS field. When the value of the status flag bit is 0, it indicates that the unit is not used and is in an idle state. The process group can be used only when the ENABLED field is 0.

[0057] like Figure 6 As shown, each process group corresponds to a first process and a second process. When a process group is locked, the port numbers of its corresponding first and second processes are also fixed, which makes it easier to find problem logs.

[0058] In one embodiment of the present invention, the number of process groups is consistent with the number of logical disks of the relay machine. When the control process uses a certain process group, the process group is locked to prevent other file import and export programs from using the process group. This ensures that only one process is reading and writing on the same disk at the same time, thereby guaranteeing the disk read and write speed and maximizing resource utilization.

[0059] Figure 3 This is a flowchart of the process group locking and unlocking process according to an embodiment of the present invention, as follows: Figure 3 As shown, in one embodiment of the present invention, the method further includes steps S301 and S302.

[0060] Step S301: After determining the currently idle process group, set the status flag corresponding to the currently idle process group to the first value in the first process control table.

[0061] In one specific embodiment of the present invention, the present invention queries whether there is a free UNIT ID in Oracle table A. If there is, the UNIT ID record is locked to prevent other control processes from using the UNIT ID.

[0062] Step S302: After calling the second process in the currently idle process group to import the target data stored in the transfer module into the second database, the status flag corresponding to the currently idle process group is set to the second value in the first process control table.

[0063] In one specific embodiment of the present invention, after the cross-database migration instruction is finally executed, the control process will unlock the corresponding UNIT ID in Oracle table A, and the task will end.

[0064] In one specific embodiment of the present invention, after the cross-database migration instruction is finally executed, the STATUS field value of the corresponding UNIT ID row in Oracle table A will be set to 0, the unlock time field will be updated, and the LOCKER_ID field value will be cleared.

[0065] When a new task process is initiated, all data records in the table with the ENBALE field (enabled status field) set to 0 and the STATUS field (status flag) set to 0 are retrieved. The first idle process group is selected in sequence and locked for use.

[0066] In one embodiment of the present invention, the data cross-database migration method of the present invention further includes:

[0067] Obtain a preset second process control table, wherein each process group corresponds to a second data record in the second process control table, and the second data record contains the IP and port information of the first process and the second process in the corresponding process group.

[0068] In one embodiment of the present invention, the second process control table may specifically be an Oracle table, and in some embodiments of the present invention, the second process control table is referred to as Oracle table B.

[0069] In one embodiment of the present invention, step S102 above, which involves calling the first process in the currently idle process group to export the target data from the first database and store it in a preset transfer module, specifically includes:

[0070] Retrieve the IP and port information of the first process in the currently idle process group from the second process control table;

[0071] Based on the IP and port information of the first process in the currently idle process group, the first process in the currently idle process group is invoked to export the target data from the first database and store it in the relay module.

[0072] In one embodiment of the present invention, step S103 above, which involves calling the second process in the currently idle process group to import the target data stored in the transfer module into the second database, specifically includes:

[0073] Retrieve the IP and port information of the second process in the currently idle process group from the second process control table;

[0074] Based on the IP and port information of the second process in the currently idle process group, the second process in the currently idle process group is invoked to import the target data stored in the relay module into the second database.

[0075] In one specific embodiment of the present invention, after the second process control table is initialized, each UNIT ID corresponds to one HIVE type process and two MPPDB type processes (one for import and one for export). Each process records the IP address and port number of the service provider. Each UNIT ID is assigned to a directory on the relay machine (each directory corresponds to a mounted disk). The directory corresponding to the relay machine is the last field WORK_DIR in this table. For example, when a request finds an idle UNIT ID from the first process control table and then finds the IP address and port information of the process corresponding to that UNIT ID from the second process control table, when data crosses the database cluster, the data will first be exported to the WORK_DIR directory corresponding to the process UNIT ID, such as / app / data2. Subsequent steps will then read data from this directory.

[0076] Figure 4 This is a flowchart illustrating the generation of a folder in the transit module according to an embodiment of the present invention, such as... Figure 4 As shown, in one embodiment of the present invention, the data cross-database migration method of the present invention further includes steps S401 to S403.

[0077] Step S401: Generate unique identifier information corresponding to the cross-database migration instruction.

[0078] Step S402: Generate a unique file name based on the unique identifier information.

[0079] In one embodiment of the present invention, a unique file name can be constructed by concatenating the unique identifier information (e.g., taskID) and time information.

[0080] Step S403: Create a folder corresponding to the unique file name in the transit module.

[0081] In one embodiment of the present invention, this step may specifically involve first determining the directory corresponding to the currently idle process group in the transfer module, and then creating a folder corresponding to the unique file name in that directory, thereby facilitating the subsequent problem location.

[0082] In one embodiment of the present invention, step S102 above, which involves calling the first process in the currently idle process group to export the target data from the first database and store it in a preset transfer module, specifically includes:

[0083] The first process in the currently idle process group is invoked to export the target data from the first database and store it in the folder.

[0084] In one embodiment of the present invention, step S103 above, which involves calling the second process in the currently idle process group to import the target data stored in the transfer module into the second database, specifically includes:

[0085] The second process in the currently idle process group is invoked to import the target data stored in the folder into the second database.

[0086] Figure 7 This is a schematic diagram of cross-database migration according to an embodiment of the present invention, as shown below. Figure 7 As shown, in a specific embodiment of the present invention, the cross-database migration of the present invention specifically includes the following steps:

[0087] 1. When the control process receives a cross-database migration instruction, it generates a unique taskID for the instruction. It then checks if a free UNIT ID exists in Oracle table A. If it does, the process locks the UNIT ID record to prevent other control processes from using it. If no free UNIT ID exists, the request is queued and waits.

[0088] 2. Depending on the requested data transfer direction, assuming the data is being transferred from the Hive database to the MPP database, the system retrieves the IP address and port information of the NODE process corresponding to the UNIT ID obtained in step 1 from Oracle table B, and sends the relevant instructions. Once the NODE process receives the instructions, it exports the data from the corresponding Hive database to the corresponding directory on the relay machine. If the data transfer direction is reversed, the system retrieves the IP address and port information of the GDS process corresponding to the UNIT ID, sends the relevant requests, and once the GDS process receives the instructions, it exports the data from the MPP database to the corresponding directory on the relay machine.

[0089] 3. After the request in step 2 is successful, the result will be sent to the control process. Assuming data is being transferred from the Hive database to the MPP database, the control process will query Oracle table B again to obtain the IP address and port of the GDS process corresponding to that UNIT ID, and then send the command. Upon receiving the command, the GDS process will load the text file in the corresponding directory of the relay machine into the MPP table.

[0090] 4. Once the request is successfully completed, the control process will unlock the corresponding UNIT ID in Oracle table A, and the task will end.

[0091] As can be seen from the above embodiments, the method of the present invention achieves at least the following beneficial effects:

[0092] 1. The data cross-database migration method provided by this invention shields the differences in underlying implementation technologies, and can migrate data from MPPDB cluster to Hadoop cluster, or from Hadoop cluster to MPPDB cluster, or from MPPDB cluster to MPPDB cluster, or from Hadoop cluster to Hadoop cluster.

[0093] 2. The method provided by this invention effectively reduces the number of network transmissions of data through the function of the relay module.

[0094] 3. The method provided by this invention can facilitate the location of the problematic process and logs for troubleshooting when problems occur during the data migration process.

[0095] 4. The modular solution provided by this invention greatly decouples the deployment complexity of the entire system. The control module, NODE process, and GDS process can all be deployed using multiple instances, thereby improving the concurrency of the entire system and greatly facilitating the expansion of cluster capabilities.

[0096] The following will explain in detail some of the terms used in this invention:

[0097] 1. The cross-database migration mentioned in this invention refers to the transfer of data between two different types of data warehouses. The two types of databases referred to in this article can be Hive database based on the Hadoop system and MPP database.

[0098] 2. Hive Database: A data warehouse based on Hadoop, with the underlying storage being the HDFS file system. It can be used to store, query, and analyze large-scale data stored in Hadoop.

[0099] 3. MPP Distributed Database: This is a distributed database based on the MPP (Massively Parallel Processing) architecture, also known as a large-scale parallel analysis database, such as Huawei's GAUSSDB product.

[0100] 4. GDS process: This method is a file import and export method based on Huawei MPPDB. It can export data from tables in MPPDB to files or load files into MPPDB.

[0101] 5. NODE process: This program is a self-developed file import and export method based on the Hadoop system. Through this process, you can connect to the Hadoop cluster to export table data from the Hive database to a file, or load a file into a table in the Hive database.

[0102] 6. Oracle tables: can be used to store relevant process information and to assist in the control of data migration processes.

[0103] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0104] Based on the same inventive concept, embodiments of the present invention also provide a data cross-database migration apparatus, which can be used to implement the data cross-database migration method described in the above embodiments, as described in the following embodiments. Since the principle by which the data cross-database migration apparatus solves the problem is similar to that of the data cross-database migration method, embodiments of the data cross-database migration apparatus can refer to embodiments of the data cross-database migration method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0105] Figure 8 This is a structural block diagram of the data cross-database migration device according to an embodiment of the present invention, as shown below. Figure 8 As shown, in one embodiment of the present invention, the data cross-database migration apparatus of the present invention includes:

[0106] The process group determination unit 1 is used to determine the currently idle process groups when receiving a cross-database migration instruction to migrate target data from the first database to the second database. Each process group includes a first process and a second process. The first process is used to export data from the first database, and the second process is used to import data into the second database.

[0107] The first processing unit 2 is used to call the first process in the currently idle process group to export the target data in the first database and store it in a preset transfer module;

[0108] The second processing unit 3 is used to call the second process in the currently idle process group to import the target data stored in the transfer module into the second database.

[0109] In one embodiment of the present invention, the process group determination unit 1 specifically includes:

[0110] The first process control table acquisition module is used to acquire a preset first process control table, wherein each process group corresponds to a first data record in the first process control table. The first data record includes a status flag bit. When the status flag bit is a first value, it indicates that the corresponding process group is in a locked state. When the status flag bit is a second value, it indicates that the corresponding process group is in an idle state.

[0111] The first process control table query module is used to determine the currently idle process group based on the current value of the status flag bit in each data record of the first process control table.

[0112] In one embodiment of the present invention, the data cross-database migration apparatus of the present invention further includes:

[0113] The process group locking unit is used to set the status flag corresponding to the currently idle process group to the first value in the first process control table after determining the currently idle process group.

[0114] In one embodiment of the present invention, the data cross-database migration apparatus of the present invention further includes:

[0115] The process group unlocking unit is used to set the status flag corresponding to the currently idle process group to the second value in the first process control table after the second process in the currently idle process group imports the target data stored in the transfer module into the second database.

[0116] In one embodiment of the present invention, the data cross-database migration apparatus of the present invention further includes:

[0117] The second process control table acquisition module is used to acquire a preset second process control table, wherein each process group corresponds to a second data record in the second process control table, and the second data record contains the IP and port information of the first process and the second process in the corresponding process group.

[0118] In one embodiment of the present invention, the first processing unit 2 includes:

[0119] The first query module is used to query the IP and port information of the first process in the currently idle process group from the second process control table;

[0120] The first process invocation module is used to invoke the first process in the currently idle process group to export the target data in the first database and store it in the relay module based on the IP and port information of the first process in the currently idle process group.

[0121] In one embodiment of the present invention, the second processing unit 3 includes:

[0122] The second query module is used to query the IP and port information of the second process in the currently idle process group from the second process control table;

[0123] The second process invocation module is used to invoke the second process in the currently idle process group according to the IP and port information of the second process in the currently idle process group to import the target data stored in the relay module into the second database.

[0124] In one embodiment of the present invention, the data cross-database migration apparatus of the present invention further includes:

[0125] A unique identifier information generation unit is used to generate unique identifier information corresponding to the cross-database migration instruction;

[0126] A unique file name generation unit is used to generate a unique file name based on the unique identifier information;

[0127] The folder creation unit is used to create a folder corresponding to the unique file name in the transit module.

[0128] In one embodiment of the present invention, the first processing unit 2 is specifically used to call the first process in the currently idle process group to export the target data in the first database and store it in the folder.

[0129] In one embodiment of the present invention, the second processing unit 3 is specifically used to call the second process in the currently idle process group to import the target data stored in the folder into the second database.

[0130] To achieve the above objectives, according to another aspect of this application, a computer device is also provided. For example... Figure 9 As shown, the computer device includes a memory, a processor, a communication interface, and a communication bus. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps in the method of the above embodiments.

[0131] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0132] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as the program units corresponding to the above-described method embodiments of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods described in the above-described method embodiments.

[0133] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] The one or more units are stored in the memory and, when executed by the processor, perform the methods described in the above embodiments.

[0135] The specific details of the aforementioned computer equipment can be understood by referring to the relevant descriptions and effects in the above embodiments, and will not be repeated here.

[0136] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed in a computer processor, implements the steps in the above-described data cross-database migration method. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0137] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data cross-database migration method.

[0138] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for data migration across databases, characterized in that, include: Upon receiving a cross-database migration instruction to migrate target data from a first database to a second database, the currently idle process groups are determined, wherein each process group includes: a first process and a second process, the first process being used to export data from the first database and the second process being used to import data into the second database; The first process in the currently idle process group is invoked to export the target data from the first database and store it in a preset transfer module; The second process in the currently idle process group is invoked to import the target data stored in the transfer module into the second database; wherein, both the first process and the second process are set in the transfer module, and the first process and the second process correspond one-to-one, and their number is consistent with the number of logical disks in the transfer module; Specifically, determining the currently idle process group includes: Obtain a preset first process control table, wherein each process group corresponds to a first data record in the first process control table, and the first data record includes a status flag bit. When the status flag bit is a first value, it indicates that the corresponding process group is in a locked state, and when the status flag bit is a second value, it indicates that the corresponding process group is in an idle state. Based on the current value of the status flag bit in each data record of the first process control table, determine the currently idle process group; After determining the currently idle process group, the status flag corresponding to the currently idle process group is set to the first value in the first process control table; After the second process in the currently idle process group imports the target data stored in the transfer module into the second database, the status flag corresponding to the currently idle process group is set to the second value in the first process control table.

2. The data cross-database migration method according to claim 1, characterized in that, Also includes: Obtain a preset second process control table, wherein each process group corresponds to a second data record in the second process control table, and the second data record contains the IP and port information of the first process and the second process in the corresponding process group; The step of calling the first process in the currently idle process group to export the target data from the first database and store it in a preset transfer module specifically includes: Retrieve the IP and port information of the first process in the currently idle process group from the second process control table; Based on the IP and port information of the first process in the currently idle process group, the first process in the currently idle process group is invoked to export the target data from the first database and store it in the relay module.

3. The data cross-database migration method according to claim 2, characterized in that, The step of invoking the second process in the currently idle process group to import the target data stored in the transfer module into the second database specifically includes: Retrieve the IP and port information of the second process in the currently idle process group from the second process control table; Based on the IP and port information of the second process in the currently idle process group, the second process in the currently idle process group is invoked to import the target data stored in the relay module into the second database.

4. The data cross-database migration method according to claim 1, characterized in that, Also includes: Generate unique identifier information corresponding to the cross-database migration instruction; A unique file name is generated based on the unique identifier information; Create a folder corresponding to the unique file name in the relay module; The step of calling the first process in the currently idle process group to export the target data from the first database and store it in a preset transfer module specifically includes: The first process in the currently idle process group is invoked to export the target data from the first database and store it in the folder.

5. The data cross-database migration method according to claim 4, characterized in that, The step of invoking the second process in the currently idle process group to import the target data stored in the transfer module into the second database specifically includes: The second process in the currently idle process group is invoked to import the target data stored in the folder into the second database.

6. The data cross-database migration method according to claim 1, characterized in that, The first process is also used to import data into the first database, and the second process is also used to export data from the second database.

7. The data cross-database migration method according to claim 1, characterized in that, The first database is a Hive database and the second database is an MPP database, or the first database is an MPP database and the second database is a Hive database, or both the first database and the second database are Hive databases, or both the first database and the second database are MPP databases.

8. A data migration device across databases, characterized in that, include: The process group determination unit is used to determine the currently idle process groups when receiving a cross-database migration instruction to migrate target data from a first database to a second database. Each process group includes a first process and a second process, wherein the first process is used to export data from the first database and the second process is used to import data into the second database. The first processing unit is used to call the first process in the currently idle process group to export the target data in the first database and store it in a preset transfer module; The second processing unit is used to call the second process in the currently idle process group to import the target data stored in the transfer module into the second database; wherein, the first process and the second process are both set in the transfer module, and the first process and the second process correspond one-to-one, and their number is consistent with the number of logical disks of the transfer module; The process group determination unit specifically includes: The first process control table acquisition module is used to acquire a preset first process control table, wherein each process group corresponds to a first data record in the first process control table. The first data record includes a status flag bit. When the status flag bit is a first value, it indicates that the corresponding process group is in a locked state. When the status flag bit is a second value, it indicates that the corresponding process group is in an idle state. The first process control table query module is used to determine the currently idle process group based on the current value of the status flag bit in each data record of the first process control table. The process group locking unit is used to set the status flag corresponding to the currently idle process group to the first value in the first process control table after determining the currently idle process group. The process group unlocking unit is used to set the status flag corresponding to the currently idle process group to the second value in the first process control table after the second process in the currently idle process group imports the target data stored in the transfer module into the second database.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images