A method for migrating data across multiple databases, a storage medium, and an electronic device.

Abstract

This invention relates to the field of data processing, and in particular to a method, storage medium, and electronic device for migrating data across multiple databases. The method includes: if the data read instruction is an SQL operation instruction, parsing the SQL operation instruction to generate operation result fields and storage location information. If the verification passes, the operation result field is used as the field to be migrated; obtaining the corresponding target receiving field to be written into the database; and establishing a mapping relationship between the field to be migrated and the target receiving field to perform data migration. After the developer inputs a data read instruction or data write instruction, this invention can automatically use the corresponding storage component to call the target field in the corresponding database based on the specific parameter information carried in the instruction, forming a field mapping relationship for data migration, facilitating subsequent data migration. This process eliminates the need for developers to develop corresponding data acquisition programs, thus improving development efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method for migrating data across multiple databases, a storage medium, and an electronic device. Background Technology

[0002] In existing technologies, when developing applications for specific business scenarios, engineers often encounter the issue of storing the business data. They typically choose different storage components based on factors such as project requirements, technology stack, performance requirements, cost budget, and team familiarity. For scenarios requiring transactional consistency and complex query capabilities, storage components suitable for storing structured data, such as MySQL, PostgreSQL, Oracle, and SQL Server, are chosen. For scenarios storing large amounts of unstructured data, such as images, videos, and backup files, storage components such as Amazon S3 and Azure Blob Storage can be selected.

[0003] Given this situation, when one business needs to use data from another, technical personnel typically need to use the corresponding storage components and develop corresponding programs to read and process the data before storing it in their own database for use—that is, data migration. If a business application involves using data from multiple other businesses or different data tables within the same business during development, a significant amount of time and effort will be spent on the relatively basic and tedious data migration development work, thereby reducing development efficiency. Summary of the Invention

[0004] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0005] According to one aspect of the present invention, a multi-database data migration method is provided, which is applied to a data migration system. The data migration system includes multiple databases and a storage component corresponding to each database. The multiple databases are used to store different business data, and the types of the multiple databases are different. Each database includes multiple data tables.

[0006] The method includes the following steps:

[0007] Obtain data read instructions;

[0008] If the data read command is an SQL operation command, the Druid library is used to parse the SQL operation command and generate a parsing result. The parsing result includes the operation result field and the storage location information corresponding to the data supply field. The storage location information includes the storage table name and the storage field name. The SQL operation command is used to indicate that a preset operation will be performed after reading the value of the corresponding data supply field in the database.

[0009] The SQL operation instructions are validated based on the storage location information.

[0010] If the verification passes, the operation result field will be used as the field to be migrated.

[0011] According to the data writing instruction, obtain the target receiving field corresponding to be written to the database;

[0012] Establish a mapping relationship between the fields to be migrated and the target receiving fields, and perform data migration based on the mapping relationship;

[0013] Verification processing includes:

[0014] Use the corresponding storage component to retrieve the original field set corresponding to the table name stored in the database;

[0015] If the stored field name is a subset of the original field set, the validation will pass.

[0016] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above-described multi-database data migration method.

[0017] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described multi-database data migration method.

[0018] The present invention has at least the following beneficial effects:

[0019] The data migration system in this invention integrates multiple databases and corresponding storage components for each database. Therefore, after developers input data read or write commands, the system can automatically use the corresponding storage component to call the target field in the corresponding database based on the specific parameter information carried in the command, forming a field mapping relationship for data migration, thus facilitating subsequent data migration. This process eliminates the need for developers to create their own data acquisition programs, improving development efficiency.

[0020] In addition, this invention also supports setting the data reading instruction as an SQL operation instruction formed by SQL statements during the acquisition of data to be migrated. This can enrich the diversity of data processing for the data to be migrated, such as performing multi-field aggregation, summation, difference and other operations. That is, the initial data can be processed before migration to improve the usability of the data. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a multi-database data migration method provided in an embodiment of the present invention.

[0023] Figure 2 This is a flowchart illustrating a data migration task execution method provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] As one possible embodiment of the present invention, such as Figure 1 As shown, a multi-database data migration method is provided, applied to a data migration system. The data migration system includes multiple databases and corresponding storage components for each database. The multiple databases are used to store different business data, and the databases differ in type. Each database includes multiple data tables.

[0026] The multi-database data migration method in this embodiment can be the specific method for the task information acquisition module in the data migration system. It is mainly used to quickly and conveniently establish the migration task information corresponding to the data migration based on the simple and necessary parameters of the migration task input by the user, so as to facilitate the execution of the subsequent migration task. Specifically, the method includes the following steps:

[0027] S100: Data Retrieval Command. This command is entered by the user based on their specific needs. Typically, this command indicates which fields in the existing database the user requires data from, i.e., specifying the fields to be migrated.

[0028] Typically, the fields to be migrated can be fields contained in a table within a database, or they can be new fields created by processing multiple different fields. Of course, these fields can also come from different tables within the same database.

[0029] Specifically, when the data reading instruction is an SQL operation instruction, multiple different fields can be processed to form new fields, thereby increasing the richness of the acquired data and thus improving the availability of the data.

[0030] The following steps, S200 to S400, describe the methods for determining the fields to be migrated when the data read instruction is an SQL operation instruction:

[0031] S200: If the data read instruction is an SQL operation instruction, the Druid library is used to parse the SQL operation instruction and generate a parsing result. The parsing result includes the operation result field and the storage location information corresponding to the data supply field. The storage location information includes the storage table name and the storage field name. The SQL operation instruction is used to indicate the preset operation to be performed after reading the value of the corresponding data supply field in the database.

[0032] The SQL operation commands mentioned above can be customized by the user according to their own needs. SQL (Structured Query Language) is a standard language for managing relational databases, capable of representing and executing various database operations. The Druid library, an open-source tool from Alibaba, provides a powerful SQL parsing module that can parse SQL statements into an Abstract Syntax Tree (AST), which is extremely useful for tasks such as SQL statement analysis, optimization, and rewriting. Using the Druid library for SQL parsing in this step allows for more accurate determination of various user-defined operation fields (i.e., operation result fields) and the corresponding storage locations of the data supply fields.

[0033] Specifically, let's take the SQL operation command for adding different fields from two different tables as an example for SQL parsing:

[0034] The SQL operation commands entered are as follows:

[0035] SELECT

[0036] TableA.common_field,(TableA.field1+TableB.field2)AS sum_of_fields

[0037] FROM

[0038] TableA

[0039] INNER JOIN

[0040] TableB ON TableA.common_field=TableB.common_field;

[0041] In this example:

[0042] INNER JOIN is used to link TableA and TableB based on the common_field field.

[0043] TableA.field1+TableB.field2 calculates the sum of these two fields in each row of the linked records.

[0044] AS sum_of_fields assigns an alias to the calculated sum, making it easier to identify in the query results.

[0045] In practical applications, users need to adjust the above SQL operation commands according to the specific table structure and relationship logic. This allows users to more flexibly analyze and process existing data, generating new data that is more meaningful and applicable.

[0046] S300: Verify SQL operation commands based on storage location information.

[0047] Verification processing includes:

[0048] S301: Use the corresponding storage component to retrieve the original field set corresponding to the table name stored in the database. The database can be a Hive, MySQL, Oracle, or Doris database.

[0049] S302: If the stored field name is a subset of the original field set, the validation passes.

[0050] S303: If the intersection of the stored field name and the original field set is empty, a default error message will be generated indicating that there is an error in the generated SQL operation command.

[0051] S400: If the verification passes, the operation result field will be used as the field to be migrated.

[0052] S300 performs SQL operation command verification based on the field attribution relationship. Since SQL operation commands are user-written commands, they may contain errors. Therefore, S300 verification can ensure the accuracy of the fields to be migrated subsequently obtained.

[0053] In addition, when the data read instruction is for locating data table information, the original standard fields in the data table can be quickly extracted by following the steps in S210 to S230, so that users can perform corresponding add or delete operations according to their own needs.

[0054] S210: If the data read instruction is data table location information, which includes database ID and data table ID, then the corresponding storage component is called according to the database ID to read the data table to be migrated corresponding to the data table ID, and the original field set of the data table to be migrated corresponding to the data table ID is generated.

[0055] To retrieve all field information from an existing table in a database, you would typically use an SQL query or a database management tool. The storage component in this step is for database management tasks.

[0056] Most database management tools (such as MySQL Workbench, Oracle SQL Developer, SQL Server Management Studio, etc.) provide a graphical interface that allows you to directly view the structure of database tables, including all field names, data types, lengths, and whether null values ​​are allowed. Simply connect to the database, select the target database and table, and you can usually see this detailed information under tabs such as "Design" or "Table Information".

[0057] S220: In response to field add / delete commands, perform add / delete operations on the corresponding fields in the original field set.

[0058] S230: Use the remaining fields as fields to be migrated.

[0059] S500: Based on the data write command, retrieve the target receiving field corresponding to the data to be written to the database. The database to be written to can be a Hive database, a MySQL database, an Oracle database, a Doris database, an HDFS database, or an Elasticsearch database.

[0060] The S500 includes:

[0061] S501: Obtain the data table location information corresponding to the target received data table. The data table location information includes: database ID and data table ID.

[0062] S502: Based on the database ID, call the corresponding storage component to read the target receiving data table corresponding to the data table ID, and generate the original field set of the target receiving data table corresponding to the data table ID.

[0063] S503: In response to field add / delete commands, perform add / delete operations on the corresponding fields in the original field set.

[0064] S504: Use the remaining fields as the target receive fields.

[0065] Since in actual data migration, it is generally not necessary to modify the data in the target receiving data table, only the fields that need to be imported need to be determined. Therefore, this process can be the same as the steps in S210 to S230.

[0066] S600: Establish a mapping relationship between the field to be migrated and the target receiving field, and perform data migration based on the mapping relationship.

[0067] The S600 includes:

[0068] S601: Get task dependency configuration information.

[0069] S602: When the conditions restricted by the task dependency configuration information are met, data migration is performed according to the mapping relationship.

[0070] Task dependency configuration information can be used to limit the execution conditions of corresponding data migration tasks. Since data migration requires retrieving data from corresponding data tables, and some data tables are updated in real time, data migration is usually performed after the data tables are updated to ensure data accuracy. These conditions constitute the task dependencies. Therefore, the following example is provided to better execute data migration tasks based on dependencies.

[0071] As another possible embodiment of the present invention, such as Figure 2 As shown, a method for executing data migration tasks is also provided, applied to a data migration system. The data migration system also includes a data migration platform and a task scheduling platform. Communication is established between the data migration platform and the task scheduling platform. The task scheduling platform is the Apache DolphinScheduler task scheduling platform.

[0072] Apache DolphinScheduler is a distributed, scalable, and visual DAG workflow task scheduling open-source system. Suitable for enterprise-level scenarios, it provides a solution for visually managing tasks, workflows, and the entire lifecycle of data processing. Apache DolphinScheduler aims to resolve complex big data task dependencies and provide applications with data and relationships in various OPS orchestrations. It addresses the problem of complex ETL (Extract, Transform, Load) dependencies in data development, making it difficult to monitor task health. DolphinScheduler assembles tasks in a DAG (Directed Acyclic Graph) streaming manner, enabling real-time monitoring of task execution status and supporting operations such as retrying, resuming failed tasks at specified nodes, pausing, resuming, and terminating tasks.

[0073] The data migration task execution method includes the following steps:

[0074] W100: The data migration platform obtains migration task configuration information. This configuration information includes data migration information and task dependency configuration information. In this embodiment, the data migration information refers to the mapping relationship established between the fields to be migrated and the target receiving fields in the previous embodiment.

[0075] The migration task configuration information is the information obtained from the above embodiments.

[0076] Specifically, the task dependency configuration information includes the dependent task ID.

[0077] W101: Obtain the task ID as follows:

[0078] W102: Obtain the dependency task ID corresponding to each newly created data migration task through the task scheduling platform.

[0079] The task scheduling platform in this implementation manages multiple tasks, including established data migration tasks and other tasks such as database data update tasks. These existing tasks typically carry dependency hints. Therefore, when a new data migration task needs to use the data from these tasks, the corresponding task dependency configuration information can be written based on the dependency hints.

[0080] Since the newly created data migration task has not yet been processed by the task scheduling platform, it has not been assigned a corresponding new task ID. Therefore, the task dependency configuration information is currently missing the new task ID.

[0081] W200: The data migration platform generates execution scripts for data migration tasks based on the acquired data migration information. These scripts then call the corresponding heterogeneous data source offline synchronization tool to perform the data migration. The heterogeneous data source offline synchronization tool is DataX. DataX is an open-source heterogeneous data source offline synchronization tool from Alibaba, dedicated to achieving stable and efficient data synchronization between various heterogeneous data sources, including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, and FTP.

[0082] Since the basic data migration configuration information in the migration task configuration information is complete, an execution script can be generated from this information. This script is used by heterogeneous data source offline synchronization tools (such as DataX) to migrate and synchronize data between different databases according to the existing data migration information.

[0083] W300: The data migration platform will send the execution script to the task scheduling platform.

[0084] W400: The task scheduling platform performs compliance checks on the executed scripts and generates compliance results.

[0085] W500: If the compliance result is that the verification is passed, the task scheduling platform will generate a new task ID for the data migration task corresponding to the migration task configuration information and send it to the data migration platform.

[0086] After generating compliance results, the method also includes:

[0087] W501: If the compliance result is a failure to pass the inspection, the task scheduling platform will generate a preset alarm message for the data migration task corresponding to the migration task configuration information and send it to the data migration platform. This preset alarm message is used to indicate that there is a problem with the corresponding execution script, which needs to be corrected and then uploaded again.

[0088] In the data migration system of this embodiment, the task scheduling platform ultimately manages these newly created data migration tasks. Therefore, the generated execution scripts need to be sent to the task scheduling platform for subsequent scheduling. Of course, in this embodiment, the task scheduling platform itself also needs to check the compliance of the execution scripts to ensure that each execution script stored on it is a normally functioning script, thus guaranteeing the normal startup of subsequent tasks. Simultaneously, if the execution script is compliant, a new task ID is generated to uniquely represent the task corresponding to that execution script, and this new task ID is returned to the data migration platform.

[0089] W600: The data migration platform generates the dependency execution instructions for the data migration task corresponding to the new task ID based on the new task ID and task dependency configuration information, and sends them to the task scheduling platform.

[0090] After receiving a new task ID, the data migration platform can use this new task ID to complete the original task dependency configuration information that contains missing information. Specifically, the new task ID is entered into the location where the ID needs to be changed in the task dependency configuration information. Then, based on the completed task dependency configuration information, the corresponding dependency execution instructions can be generated.

[0091] W700: The task scheduling platform executes instructions based on dependencies. When the corresponding dependency conditions are met, it runs the execution script to call the corresponding heterogeneous data source offline synchronization tool to perform data migration.

[0092] In this embodiment, the execution script and dependency execution instructions corresponding to the new data migration task can be imported into the task scheduling platform. The task scheduling platform then executes the instructions according to the dependency relationships. When the corresponding dependency conditions are met, the execution script runs to call the corresponding heterogeneous data source offline synchronization tool to perform the data migration. Therefore, this embodiment can add corresponding dependencies to any newly created data migration task, so as to more accurately and meticulously restrict the execution conditions of the task, ultimately improving the accuracy of the data obtained during data migration.

[0093] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0094] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0095] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0096] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0097] An electronic device according to this embodiment of the invention. The electronic device is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the invention.

[0098] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0099] The memory stores program code that can be executed by a processor, causing the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of the present invention.

[0100] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0101] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0102] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0103] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0104] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0105] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0106] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0109] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0110] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0111] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0112] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-database data migration method, characterized by, The application is applied to a data migration system, the data migration system comprises a plurality of databases and a storage component corresponding to each database, the plurality of databases are used for storing different business data, and the types of the plurality of databases are different; each database comprises a plurality of data tables; The method comprises the following steps: obtaining a data reading instruction; if the data reading instruction is an SQL operation instruction, performing SQL analysis on the SQL operation instruction by using a Druid library to generate an analysis result; the analysis result comprises a storage position information corresponding to an operation result field and a data supply field, and the storage position information comprises a storage table name and a storage field name; the SQL operation instruction is used to represent a preset operation after reading the value of the corresponding data supply field in the reading database; performing a check processing on the SQL operation instruction according to the storage position information; if the check passes, the operation result field is used as a to-be-migrated field; obtaining a target receiving field corresponding to the to-be-migrated field in a writing database according to a data writing instruction; establishing a mapping relationship between the to-be-migrated field and the target receiving field, and performing data migration according to the mapping relationship; the check processing comprises: obtaining an original field set corresponding to the storage table name in the reading database by using the corresponding storage component; if the storage field name is a subset of the original field set, the check passes; The data migration system further comprises a data migration center and a task scheduling platform; the data migration center and the task scheduling platform are in communication connection; The method further comprises the following steps: W100: the data migration center obtains migration task configuration information; the migration task configuration information comprises data migration information and task dependency configuration information; W200: the data migration center generates an execution script corresponding to a data migration task according to the obtained data migration information; the execution script is used to call a corresponding heterogeneous data source offline synchronization tool to perform data migration work; W300: the data migration center sends the execution script to the task scheduling platform; W400: the task scheduling platform performs compliance detection on the execution script to generate a compliance result; W500: if the compliance result passes the test, the task scheduling platform generates a new task ID of the data migration task corresponding to the migration task configuration information and sends it to the data migration center; W600: the data migration center generates a dependency relationship execution instruction of the data migration task corresponding to the new task ID according to the new task ID and the task dependency configuration information, and sends it to the task scheduling platform; W700: the task scheduling platform runs the execution script to call the corresponding heterogeneous data source offline synchronization tool to perform data migration work according to the dependency relationship execution instruction when the corresponding dependency condition is met.

2. The method of claim 1, wherein, After obtaining the original field set corresponding to the storage table name in the reading database, the method further comprises: if the intersection of the storage field name and the original field set is empty, a preset prompt information indicating that the SQL operation instruction has an error is generated.

3. The method of claim 1, wherein, The reading database is a hive database, a mysql database, an oracle database or a doris database.

4. The method of claim 1, wherein, The writing database is a hive database, a mysql database, an oracle database, a doris database, an hdfs database, or an elasticsearch database.

5. The method of claim 1, wherein, The data table positioning information includes a database ID and a data table ID. After the data reading instruction is obtained, the method further includes: If the data reading instruction is data table positioning information, a corresponding storage component is called according to the database ID to read the to-be-migrated data table corresponding to the data table ID, and an original field set of the to-be-migrated data table corresponding to the data table ID is generated; In response to a field addition or deletion instruction, a corresponding field in the original field set is added or deleted; The remaining fields are taken as to-be-migrated fields.

6. The method of claim 1, wherein, According to the data writing instruction, a corresponding target receiving field in the writing database is obtained, including: Data table positioning information corresponding to the target receiving data table is obtained; the data table positioning information includes a database ID and a data table ID; A corresponding storage component is called according to the database ID to read the target receiving data table corresponding to the data table ID, and an original field set of the target receiving data table corresponding to the data table ID is generated; In response to a field addition or deletion instruction, a corresponding field in the original field set is added or deleted; The remaining fields are taken as target receiving fields. 7.A non-transitory computer-readable storage medium storing a computer program, and the computer program comprises the following steps of, The computer program, when executed by a processor, implements a multi-database data migration method according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor, when executing the computer program, implements a multi-database data migration method according to any one of claims 1 to 6.

Images