A method, system, device and storage medium for data migration between databases

Abstract

The present invention discloses a method, system, device and storage medium for data migration between databases. The method comprises: initializing and configuring a source database and a target database using a graphical user interface; dynamically switching to a source database or a target database using a dynamic data source routing mechanism; performing metadata conversion according to field type differences of metadata in the source database and the target database; and performing asynchronous multi-threaded data migration according to database migration tasks. The technical solution in the embodiment of the present invention provides a solution for mutual migration between different databases, which can flexibly switch different data sources, realize migration work between any two databases, and realize more efficient and convenient database migration.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, system, device and storage medium for data migration between databases. Background Technology

[0002] With the rapid development of information technology, databases, as a core component of data storage and management, are constantly increasing in both type and quantity. However, differences in data formats and storage methods between different databases pose significant challenges to data migration.

[0003] Traditional database migration tools are often limited in function, cumbersome in operation, and do not support the migration of spatial geographic data, thus failing to meet the diverse needs of database migration in modern information, especially in the field of geographic information (GIS).

[0004] While existing database migration systems have achieved some success, several problems and shortcomings remain. Firstly, regarding migration efficiency, as data volumes continue to increase, migration speed becomes a key concern for users. Secondly, due to the diversity of database types, the compatibility of different data types needs continuous improvement and refinement over long-term use to achieve a "the more you use it, the better it becomes" effect. Summary of the Invention

[0005] This invention provides a method, system, device, and storage medium for data migration between databases, enabling more efficient and convenient database migration.

[0006] In a first aspect, embodiments of the present invention provide a data migration method between databases, comprising:

[0007] Initialize and configure the source and target databases through a graphical user interface;

[0008] A dynamic data source routing mechanism is used to dynamically switch to the source database or the target database;

[0009] Metadata conversion is performed based on the differences in field types between the source and target databases.

[0010] Asynchronous multi-threaded data migration is performed based on the database migration task.

[0011] Secondly, embodiments of the present invention also provide a data migration method between databases, the method comprising:

[0012] Initialize and configure the source and target databases through a graphical user interface;

[0013] A dynamic data source routing mechanism is used to dynamically switch to the source database or the target database;

[0014] Identify the data source interfaces of different databases;

[0015] Obtain metadata from the source and target databases;

[0016] Perform field type conversion between the source and target databases;

[0017] Asynchronous multi-threaded data migration is performed based on the database migration task.

[0018] Thirdly, embodiments of the present invention also provide a data migration method between databases, the method comprising:

[0019] Initialize and configure the source and target databases through a graphical user interface;

[0020] A dynamic data source routing mechanism is used to dynamically switch to the source database or the target database;

[0021] Metadata conversion is performed based on the differences in field types between the source and target databases.

[0022] Generate and execute the table creation script, and use multiple threads to retrieve data from the source database;

[0023] Transform the data in the source database and write the data to the target database asynchronously;

[0024] Obtain source database index metadata, generate target database index script and execute it;

[0025] Obtain constraint metadata, generate the target database constraint script, and execute it.

[0026] Fourthly, embodiments of the present invention also provide a data migration system between databases, configured to implement any of the methods of the embodiments of the present invention, the system comprising:

[0027] The initialization configuration module is used to initialize and configure the source and target databases through a graphical user interface.

[0028] The dynamic switching module is used to dynamically switch to the source database or the target database using a dynamic data source routing mechanism.

[0029] The metadata conversion module is used to convert metadata based on the differences in field types between the source and target databases.

[0030] The data migration module is used to perform asynchronous multi-threaded data migration based on database migration tasks.

[0031] Fifthly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0032] At least one processor; and

[0033] A memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the data migration method between databases as described in any embodiment of the present invention.

[0035] Sixthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the data migration method between databases as described in any embodiment of the present invention.

[0036] This invention initializes and configures the source and target databases through a graphical user interface, then uses a dynamic data source routing mechanism to dynamically switch to the source or target database. Next, it performs metadata conversion based on the differences in the field types of metadata in the source and target databases. Finally, it performs asynchronous multi-threaded data migration based on the database migration task. This not only allows for the selection of the appropriate data source based on the task context and flexible handling of migration tasks between different data sources, but also enables more efficient and convenient database migration. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a data migration method between databases provided in Embodiment 1 of the present invention;

[0039] Figure 2 This is a flowchart of a data migration method between databases provided in Embodiment 2 of the present invention;

[0040] Figure 3 This is a flowchart of a data migration method between databases provided in Embodiment 3 of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of a data migration system between databases provided in Embodiment 4 of the present invention;

[0042] Figure 5This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0044] Example 1

[0045] Figure 1 This is a flowchart of a data migration method between databases provided in Embodiment 1 of the present invention. This embodiment is applicable to data migration between any different databases. The method can be executed by the data migration system between databases (hereinafter referred to as the data migration system) in this embodiment of the present invention. It can efficiently realize functions such as data migration, incremental migration, comparison, scheduled backup, single backup, and batch export of scripts between common databases. The system can be implemented in software and / or hardware. It can be run by combining a JavaFX visual interface with a Spring Boot backend service and integrated into a computer device, which can be a mobile terminal or a server.

[0046] like Figure 1 As shown, the method specifically includes the following steps:

[0047] S110. Initialize and configure the source and target databases through the graphical user interface.

[0048] The initialization and configuration process includes selecting and configuring the connection information for the source and target databases, as well as configuring data migration parameters.

[0049] Connection information includes: database type, address, port, username, and password, etc.

[0050] Data migration parameters include: the table name to be migrated, field mapping rules, migration batch size, etc.

[0051] The configuration of field mapping rules will be initialized after the user selects the source database and the target database. This includes creating a new migration configuration table and a migration mapping table, and adding mapping data to the migration mapping table. If the current database has already been initialized, it will not be initialized again.

[0052] It's important to note that field mapping rules refer to the correspondence between field types in metadata from different databases. For example: mapping the field type "varchar2" in Oracle to the field type "varchar" in PostgreSQL; mapping the field type "blob" in Oracle to the field type "bytea" in PostgreSQL; mapping the field type "ado_geometry" in Oracle to the field type "geometry" in PostgreSQL; mapping the field type "clob" in Oracle to the field type "text" in PostgreSQL; mapping the field type "nvarchar2" in Oracle to the field type "varchar" in PostgreSQL; and so on.

[0053] In this embodiment of the invention, both the source database and the target database can be databases commonly used in the relevant technical field, such as GaussDB, Oracle, PostgreSQL, MySQL, etc. The source and target databases can be homogeneous (e.g., multiple MySQL databases) or heterogeneous (e.g., a mixture of MySQL and Oracle). Before migrating the database, initialization and configuration are required, a process completed by a graphical user interface (GUI). A GUI is a user interface for a computer operating environment that uses a graphical representation, employing windows, menus, buttons, etc., to facilitate user operation.

[0054] The purpose of initializing the source and target databases is to facilitate the data migration system in finding the source and target databases, establish data channels between the databases, and configure migration rules. The data migration system can automatically read and parse the database connection information and data migration parameters configured by the user, and then dynamically load the database metadata.

[0055] S120: Employs a dynamic data source routing mechanism to dynamically switch to the source database or the target database.

[0056] It's important to note that dynamic data source routing is a technology implemented at the database access layer that dynamically selects a data source for data access based on specific conditions. This mechanism is particularly important in multi-tenant systems and microservice architectures because it can meet the data isolation requirements of different tenants or service instances.

[0057] The implementation process of dynamic data source routing includes: routing strategy formulation, data source configuration, route selection, and data access. Routing strategy formulation refers to defining a routing strategy to determine which data source to select under what conditions. These conditions may include user identity, request parameters, system configuration, etc. Data source configuration refers to configuring multiple data sources in the system, each corresponding to a specific database or database instance. Route selection refers to determining which data source to select for data access based on the preset routing strategy when the system receives a data access request; this step is usually completed automatically by routing middleware or a framework. Data access refers to performing data access operations, such as querying, inserting, and updating, through the selected appropriate data source.

[0058] In this embodiment of the invention, the data migration system employs a dynamic data source routing mechanism, which can dynamically switch to the source database or the target database according to the needs of the current migration task. This mechanism configures multiple data sources and selects the appropriate data source at runtime based on the task context, thereby enabling flexible handling of migration tasks between different data sources.

[0059] S130. Perform metadata conversion based on the differences in field types between the source and target databases.

[0060] Metadata includes: table structure, field types, indexes, etc.

[0061] It should be noted that metadata mainly describes information about data attributes, which is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0062] In this embodiment of the invention, the data migration system pre-configures the mapping data in the migration mapping table according to the correspondence between field types of metadata in different databases. When the metadata field type of the source database is queried, the field type to be converted can be found in the migration mapping table according to the field mapping rules.

[0063] S140. Perform asynchronous multi-threaded data migration based on the database migration task.

[0064] Specifically, before performing asynchronous multi-threaded data migration, data conversion is required. The converted data is then written to the target database using multi-threading technology and asynchronous methods to avoid blocking the main thread and improve system response speed.

[0065] The technical solution of this invention initializes and configures the source and target databases through a graphical user interface, then uses a dynamic data source routing mechanism to dynamically switch to the source or target database. Next, it performs metadata conversion based on the differences in the field types of metadata in the source and target databases. Finally, it performs asynchronous multi-threaded data migration according to the database migration task. That is, this application first performs metadata conversion and then performs data migration. Users can easily achieve database migration without a cumbersome configuration and installation process, which significantly improves the speed and efficiency of data migration.

[0066] Furthermore, during asynchronous multi-threaded data migration, migration logs are recorded in real time. These logs include: the start and end times of the migration task, the amount of data migrated, and error messages encountered during the migration process.

[0067] The data migration system records migration logs in real time, including the start and end times of migration tasks, the amount of data migrated, and error messages encountered, so that users can perform subsequent analysis and troubleshooting.

[0068] Example 2

[0069] Figure 2 This is a flowchart illustrating a data migration method between databases provided in Embodiment 2 of the present invention. This embodiment is based on Embodiment 1 and further optimized. This method can be executed by a data migration system. Figure 2 As shown, the data migration method between databases provided in this embodiment specifically includes the following steps:

[0070] S210 initializes and configures the source and target databases through a graphical user interface.

[0071] Users select and configure connection information for the source and target databases through the graphical user interface (GUI) provided by JavaFX, including database type, address, port, username, password, etc. One of the purposes of configuring connection information is to facilitate searching for different databases based on the database information.

[0072] In addition, users need to configure data migration parameters, such as the table name to be migrated, field mapping rules, and migration batch size. One of the purposes of configuring data migration parameters is to set metadata transformation rules.

[0073] After initial configuration, the system automatically reads and parses the database connection information configured by the user, and dynamically loads the database metadata, including table structure, field types, indexes, etc.

[0074] S220 employs a dynamic data source routing mechanism, dynamically switching to the source or target database.

[0075] The data migration system employs a dynamic data source routing mechanism, which dynamically switches to the source or target database using DynamicDataSourceContextHolder based on the needs of the current migration task.

[0076] This mechanism enables flexible handling of migration tasks between different data sources by configuring multiple data sources and selecting the appropriate data source at runtime based on the task context.

[0077] S230 identifies the data source interface for different databases.

[0078] During the initial configuration phase, the data migration system has already set the database ports, determined the source and target databases according to the needs of the current migration task, and then determined the data source interfaces for different databases based on the initial configuration.

[0079] S240: Obtain metadata from the source and target databases.

[0080] In this embodiment of the invention, the source database is first queried. Taking Oracle as the source database and PostgreSQL as the target database as an example, the query process is as follows: the geometry type data of the Oracle source database is queried through MyBatis's custom dynamic SQL. Here, when the table is created, geom is the default name of the spatial field (the name can be customized). The application encapsulates the data to convert it into WKT text. The Oracle database needs to use the pre-written WKT conversion function package, which is provided in the same folder.

[0081] Then, based on the conversion function and corresponding coordinate system of the target PostgreSQL database, the WKT format string is converted into a geometric object and stored in the target database.

[0082] In this embodiment of the invention, after the data migration system identifies the data source interfaces of different databases, the data migration system obtains metadata, including table structure, field types, indexes, etc., and views the field types of the metadata.

[0083] S250 performs field type conversion between the source and target databases.

[0084] The data migration system performs necessary field type conversions based on the differences in field types between the source and target databases. Specifically, the system converts the field types of metadata from the source database to the target database according to field mapping rules. For example, if a metadata field in the source database Oracle is of type "varchar2", when migrating it to the target database PostgreSQL, its field type in PostgreSQL will be "varchar" according to the field mapping rules.

[0085] S260 performs asynchronous multi-threaded data migration based on the database migration task.

[0086] Asynchronous multithreaded data migration processes can utilize thread pool technology. A thread pool is a multithreaded concurrent processing technique designed to reduce the time and resources consumed in creating and destroying threads. By pre-creating and maintaining a set of threads, a thread pool can quickly allocate threads to execute tasks as needed, thereby improving system responsiveness and throughput.

[0087] The asynchronous multithreaded data migration process specifically includes: S2601, thread initialization: When the thread pool starts, it creates and initializes a certain number of threads, which are in a waiting state, ready to execute tasks at any time; S2602, task submission: When a new task needs to be executed, it is submitted to the task queue in the thread pool; S2603, thread allocation: An idle thread in the thread pool takes a task from the task queue and executes it. If all threads are busy, new tasks will wait in the task queue until a thread becomes idle; S2604, thread recycling: After the task is completed, the thread is not immediately destroyed, but returns to the thread pool to continue waiting for new tasks.

[0088] The thread pool dynamically adjusts the number of threads based on the configured strategy (such as idle time, maximum number of threads, etc.).

[0089] This invention initializes and configures the source and target databases through a graphical user interface. It employs a dynamic data source routing mechanism to dynamically switch between the source and target databases, identifies the data source interfaces of different databases, obtains metadata from both the source and target databases, performs field type conversions on both databases, and conducts asynchronous multi-threaded data migration based on the database migration task. By using pre-configured initialization information to convert field types between the source and target databases and introducing thread pool technology for asynchronous multi-threaded data migration, the speed and efficiency of data migration are significantly improved.

[0090] Example 3

[0091] Figure 3This is a flowchart illustrating a data migration method between databases provided in Embodiment 3 of the present invention. This embodiment is also based on Embodiment 1 with further optimizations. This method can be executed by a data migration system between databases. Figure 3 As shown, the data migration method between databases provided in this embodiment specifically includes the following steps:

[0092] S310 initializes and configures the source and target databases through a graphical user interface.

[0093] S320 employs a dynamic data source routing mechanism, dynamically switching to the source or target database.

[0094] S330 performs metadata conversion based on the differences in field types between the source and target databases.

[0095] In this embodiment of the invention, the data migration system supports the dynamic addition of multiple data sources and can flexibly switch between different data sources to achieve migration between any two databases.

[0096] S340 generates and executes a table creation script, using multi-threading to retrieve data from the source database.

[0097] Specifically, the table creation script is generated and executed, and multi-threaded data is retrieved from the source database. This includes splitting the migration task into multiple subtasks, with each subtask responsible for migrating a portion of the data.

[0098] Understandably, in order to improve the efficiency of data migration, the system adopted multi-threading technology to break down the migration task into multiple sub-tasks and process them in parallel.

[0099] Each subtask is responsible for migrating a portion of the data. The system manages the execution of these subtasks through a thread pool to ensure efficient resource utilization and load balancing.

[0100] S350 performs data transformation on the source database and asynchronously writes the data to the target database.

[0101] When performing data conversion on the source database, if the metadata in the source database is spatial geographic data, then the spatial geographic data will undergo format conversion, data verification, and data repair.

[0102] For spatial geographic data (such as GIS data), the data migration system provides specialized conversion algorithms and storage strategies to ensure that the data maintains its original spatial structure and accuracy during the migration process, including format conversion (such as converting one GIS format to another), data verification (checking the integrity and consistency of the data), and data repair (repairing damaged or incomplete data).

[0103] These processing steps ensure that the migrated spatial geographic data retains a usable format and accuracy in the target database.

[0104] This invention aims to provide a solution for migrating between different databases, specifically addressing the pain point of complex spatial geographic data migration in the geographic information technology industry. By optimizing the migration algorithm and enhancing data type compatibility, this invention enables more efficient and convenient database migration, while supporting modular and customizable database operation functions.

[0105] S360 retrieves source database index metadata, generates target database index script, and executes it.

[0106] The database migration work is not finished after the metadata transformation and data writing to the target database. It is also necessary to create an index for the metadata in the target database.

[0107] Build the target database index script, including: querying index metadata and processing index metadata.

[0108] First, based on the defined interface, an implementation class object is created using the factory pattern, and the method to retrieve index metadata is called to query the index metadata.

[0109] Then, after obtaining the index metadata of the current table, iterate through the collection, concatenate the index creation script that conforms to the target library's syntax type, write the script to a local file, and call the execution method.

[0110] S370: Obtain constraint metadata, generate target database constraint scripts, and execute them.

[0111] The process of building the target database constraint script is similar to that of indexing. First, the metadata information of the foreign key constraint in the current table is queried. Then, this information is processed to conform to the target database and the foreign key syntax rules are constructed. The script is written to the local storage and executed to achieve the migration of foreign keys.

[0112] Foreign keys, due to their special nature, must be processed last, so they are the final step in the migration process.

[0113] The embodiments of the present invention can also be applied to various scenarios that require database migration and data management, such as data backup and recovery, data migration and upgrade, data synchronization and integration, etc.

[0114] This invention initializes and configures the source and target databases through a graphical user interface. It employs a dynamic data source routing mechanism to dynamically switch between the source and target databases. Based on the differences in field types of metadata in the source and target databases, it performs metadata conversion, generates and executes table creation scripts, uses multi-threading to acquire data from the source database, performs data conversion on the source database data, asynchronously writes the data to the target database, acquires source database index metadata, generates and executes target database index scripts, acquires constraint metadata, generates and executes target database constraint scripts, and supports multiple database types and data formats. It can flexibly handle migration tasks between different data sources, and the use of multi-threading and asynchronous processing technologies significantly improves the speed and efficiency of data migration. Furthermore, special processing is applied to spatial geographic data types to ensure data integrity and accuracy.

[0115] Example 4

[0116] Figure 4 This is a schematic diagram of a data migration system between databases provided in an embodiment of the present invention. The data migration system is configured to implement the methods described in any embodiment or claim of the present invention. This embodiment is applicable to data migration between any different databases. The system can be implemented in software and / or hardware, and can run using a combination of a JavaFX visual interface and a Spring Boot backend service, and can be integrated into a computer device. The system can be integrated into any device that provides data migration functionality between databases, such as... Figure 4 As shown, the data migration system between databases specifically includes:

[0117] The initialization configuration module 410 is used to initialize and configure the source database and the target database through a graphical user interface;

[0118] The dynamic switching module 420 is used to dynamically switch to the source database or the target database using a dynamic data source routing mechanism.

[0119] Metadata conversion module 430 is used to convert metadata based on the differences in field types between the source database and the target database.

[0120] The data migration module 440 is used to perform asynchronous multi-threaded data migration based on the database migration task.

[0121] Metadata includes at least: table structure, field types, and indexes.

[0122] Optionally, the data migration module 440 is also used to: record migration logs in real time during asynchronous multi-threaded data migration.

[0123] The migration log includes: the start time and end time of the migration task, the amount of data migrated, error messages encountered during the migration process, etc.

[0124] Optionally, the initialization configuration module 410 is also used to: select and configure the connection information of the source database and the target database, and configure data migration parameters.

[0125] The connection information includes: database type, address, port, username, and password, etc.

[0126] Data migration parameters include: the table name to be migrated, field mapping rules, migration batch size, etc.

[0127] Optionally, the metadata transformation module 430 is also used for:

[0128] Identify the data source interfaces of different databases;

[0129] Obtain metadata from the source and target databases;

[0130] Perform field type conversion between the source and target databases.

[0131] Optionally, the data migration module 440 is also used for:

[0132] Generate and execute the table creation script, and use multiple threads to retrieve data from the source database;

[0133] Transform the data in the source database and write the data to the target database asynchronously;

[0134] Obtain source database index metadata, generate target database index script, and execute it;

[0135] Obtain constraint metadata, generate the target database constraint script, and execute it.

[0136] This includes generating and executing table creation scripts, and using multi-threading to retrieve data from the source database. This involves splitting the migration task into multiple subtasks, with each subtask responsible for migrating a portion of the data.

[0137] When performing data conversion on the source database, if the data in the source database is spatial geographic data, then the spatial geographic data will undergo format conversion, data verification, and data repair, etc.

[0138] In addition, the data migration system supports modular customization functions, enabling database comparison, backup, scheduled backup, script export, database synchronization, and incremental migration, making the system more flexible and scalable.

[0139] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0140] This invention provides a data migration system with an extensible interface. Through a JavaFX visual interface and modular function configuration, users can easily migrate databases without cumbersome configuration and installation processes. The system also supports modular customization of database-related operations, such as database comparison, backup, scheduled backup, script export, database synchronization, and incremental migration. These functions meet diverse user needs for database operations. The system adopts an interface-oriented design, achieving scalability and reusability. Furthermore, users can add new database types and migration algorithms as needed to adapt to evolving requirements.

[0141] Example 5

[0142] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0143] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0144] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0145] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as implementing the data migration method between databases provided in the above embodiments of the present invention:

[0146] Initialize and configure the source and target databases through a graphical user interface;

[0147] A dynamic data source routing mechanism is used to dynamically switch to the source database or the target database;

[0148] Metadata conversion is performed based on the differences in field types between the source and target databases.

[0149] Based on the database migration task, perform asynchronous multi-threaded data migration;

[0150] The metadata includes at least: table structure, field types, and indexes.

[0151] In some embodiments, the inter-database data migration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the inter-database data migration method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the inter-database data migration method by any other suitable means (e.g., by means of firmware).

[0152] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0156] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0157] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0158] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0159] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for data migration between databases, characterized in that: include: Initialize and configure the source and target databases through a graphical user interface; Adopt dynamic data source routing mechanism to dynamically switch to source database or target database; Perform metadata conversion based on the differences in metadata field types between the source database and the target database; Perform asynchronous multi-threaded data migration based on database migration tasks.

2. The method according to claim 1, characterized in that The method further comprises: When performing asynchronous multi-threaded data migration, a migration log is recorded in real time; the migration log includes: at least one of the start time and end time of the migration task, the amount of data migrated, and error information encountered during the migration process.

3. The method according to claim 1, characterized in that The initialization and configuration include: selecting and configuring the connection information of the source database and the target database, and configuring the data migration parameters; The connection information includes: at least one of: database type, address, port, user name and password; The data migration parameters include: at least one of a migrated table name, a field mapping rule, and a migration batch size.

4. The method according to claim 1, characterized in that: The metadata conversion is performed according to the difference in field types of metadata in the source database and the target database, including: Identify data source interfaces for different databases; Get metadata of source and target databases; Convert the field types of the source and target databases.

5. The method according to claim 1, characterized in that The asynchronous multi-threaded data migration is performed according to the database migration task, including: Generate and execute table creation scripts to obtain source database data through multi-threading; Perform data conversion on the source database and write the data asynchronously to the target database; Obtain the source database index metadata, generate the target database index script and execute it; Obtain constraint metadata, generate target database constraint scripts, and execute them.

6. The method according to claim 5, characterized in that The generating and executing of the table creation script to obtain source database data through multi-threading includes: Split the migration task into multiple subtasks, each of which is responsible for migrating a portion of the data.

7. The method according to claim 5, characterized in that When performing data conversion on the source database data, if the data in the source database is spatial geographic data, format conversion, data verification and data repair are performed on the spatial geographic data.

8. A data migration system between databases, characterized in that: The system is configured to implement the method according to any one of claims 1 to 7, and the system comprises: An initialization configuration module, used to initialize and configure the source database and the target database through a graphical user interface; A dynamic switching module is used to dynamically switch to a source database or a target database using a dynamic data source routing mechanism; A metadata conversion module is used to convert metadata according to the field type differences between the source database and the target database; The data migration module is used to perform asynchronous multi-threaded data migration according to the database migration tasks.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for migrating data between databases according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for migrating data between databases according to any one of claims 1 to 7 when executed.

Images