Data migration method, device and equipment, computer storage medium and program product

Abstract

The invention discloses a data migration method and device, equipment, a computer storage medium and a program product, and relates to the technical field of big data, the method comprises the steps that firstly, input information of a user is received, the input information comprises a configuration file, and the configuration file comprises connection information of a source database and a target database and a data mapping rule; respectively establishing communication connections with the source database and the target database based on the connection information of the source database and the target database; under the condition that the migration data of the source database is obtained, the data type and the data format of the migration data are converted into the data type and the data format of a target database according to a data mapping rule, and target migration data are obtained; and sending the target migration data to the target database. According to the embodiment of the invention, the automatic conversion of the data format and type is realized based on the data mapping rule, the data migration between different databases is realized, and the universality and flexibility of the data migration method are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of big data, and particularly relates to a data migration method and device, equipment, computer storage medium and program product. BACKGROUND

[0002] With the rapid development of the field of big data, the demand for exponential explosive data migration of data volume is increasingly urgent, and the demand for data migration is increasingly urgent under this background.

[0003] The prior art often uses a preset migration script or migration tool to realize data migration, but these migration tools are usually designed for specific database systems and can only support specific operating systems or database environments, such as Oracle Data Pump, which only supports data migration between Oracle databases, and SQL Server Integration Services (SSIS), which only supports data migration of database systems in SQL Server environment. Therefore, the data migration method of the prior art can only perform data migration between the same database system or specific different database systems, and cannot realize general data migration across systems, and the flexibility and universality of the data migration method are poor. SUMMARY

[0004] The embodiments of the present application provide a data migration method, device, equipment, computer storage medium and program product to solve the problem of poor flexibility and universality of the prior art data migration method.

[0005] In a first aspect, the embodiments of the present application provide a data migration method, which comprises:

[0006] Receiving input information of a user, the input information comprising a configuration file, the configuration file comprising connection information of a source database and a target database, and a data mapping rule;

[0007] Based on the connection information of the source database and the target database, a communication connection with the source database and the target database is established respectively;

[0008] In the case of obtaining migration data of the source database, the data type and data format of the migration data are converted into the data type and data format of the target database according to the data mapping rule, and target migration data is obtained;

[0009] Sending the target migration data to the target database.

[0010] In a second aspect, the embodiments of the present application provide a data migration device, which comprises:

[0011] The receiving module is configured to receive input information of a user, the input information comprising a configuration file, the configuration file comprising connection information of a source database and a target database, and a data mapping rule;

[0012] The connection module is configured to establish a communication connection with the source database and the target database respectively based on the connection information of the source database and the target database;

[0013] The conversion module is configured to, in a case where the migration data of the source database is acquired, convert a data type and a data format of the migration data into a data type and a data format of the target database according to the data mapping rule, to obtain target migration data;

[0014] The sending module is configured to send the target migration data to the target database.

[0015] In a third aspect, an embodiment of the present application provides a terminal device, which comprises a processor and a memory storing computer program instructions; and the processor implements the data migration method according to the first aspect when executing the computer program instructions.

[0016] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions; and the computer program instructions are executed by a processor to implement the data migration method according to the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, instructions in the computer program product being executed by a processor of an electronic device to cause the electronic device to perform the data migration method according to the first aspect.

[0018] The data migration method provided by the embodiment of the present application first receives input information of a user, the input information comprising a configuration file, the configuration file comprising connection information of a source database and a target database, and a data mapping rule; establishes a communication connection with the source database and the target database respectively based on the connection information of the source database and the target database; in a case where migration data of the source database is acquired, converts a data type and a data format of the migration data into a data type and a data format of the target database according to the data mapping rule, to obtain target migration data; and sends the target migration data to the target database. The connection information of the source database and the target database and the data mapping rule are managed through the configuration file, and different database types can be adapted without modifying codes by switching the configuration file; the data format and the type are automatically converted based on the data mapping rule, which guarantees the consistency of the data format and the type of the target migration data and the target database; and finally the target migration data is sent to the target database, which improves the versatility and flexibility of the data migration method. BRIEF DESCRIPTION OF DRAWINGS

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the structure of a data migration system applied to a data migration method provided in an embodiment of this application.

[0021] Figure 2 This is a flowchart illustrating the data migration method provided in an embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating one implementation of the data migration method provided in this application embodiment;

[0023] Figure 4 This is a flowchart illustrating one implementation method of migration data conversion provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the data migration device provided in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0026] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0027] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0028] Existing data migration methods often rely on the database's built-in migration tools. For example, Oracle Data Pump only supports data migration between Oracle databases, and SQL Server Integration Services (SSIS) only supports data migration between database systems within a SQL Server environment. These tools can migrate data between different instances of the same database system. However, existing tools only work on specific operating systems or database environments, typically supporting only certain database systems. This limitation means that when enterprises face migration needs across database systems, such as from Oracle to SQL Server, or from SQL Server to MySQL or other database systems, these tools cannot directly provide solutions. They also lack broad support for multiple database types and the flexibility for cross-platform operation.

[0029] To address the shortcomings of existing technologies, this application provides a data migration method. First, it receives user input information, including a configuration file. The configuration file contains connection information between a source database and a target database, as well as data mapping rules. Based on the connection information, communication connections are established with both the source and target databases. Upon obtaining migration data from the source database, the data type and format of the migration data are converted to the data type and format of the target database according to the data mapping rules, resulting in target migration data. Finally, the target migration data is sent to the target database. By managing the connection information and data mapping rules with the source and target databases through the configuration file, different database types can be adapted without code modification by switching configuration files. Automated conversion of data format and type based on data mapping rules ensures consistency between the target migration data and the target database in terms of data format and type. Finally, the target migration data is sent to the target database, improving the versatility and flexibility of the data migration method.

[0030] First, with reference to the accompanying drawings, we will introduce a data migration system applied to a data migration method provided in the embodiments of this application.

[0031] Figure 1 An embodiment of this application provides a data migration system applied to a data migration method. For example... Figure 1 As shown, the system may include: a client device 101 and a server device 102. The client device 101 is responsible for managing various configuration information during the data migration process, enabling users to interact with the system intuitively. The server device 102 is the actual executor of the data migration task. Based on the configuration information sent by the client device, it is responsible for completing a series of data migration operations such as data extraction, transformation, and loading.

[0032] The client device 101 may include a user interface module 1011 and a configuration management module 1012, while the server device 102 may include a database connection module 1021, a file import module 1022, a data conversion module 1023, a data extraction module 1024, and a data output module 1025.

[0033] The user interface module 1011 provides an interactive interface for users to facilitate data migration operations. Users can intuitively view the progress and status of the data migration through this interface, and can configure and manage the data migration task, such as setting connection information for the data source and target database. The configuration management module 1012 is responsible for managing various configuration information during the data migration process, such as database connection parameters and migration task options. It provides users with a centralized mechanism for managing and modifying configurations, ensuring that the data migration task proceeds according to user requirements. The configuration management module can read and parse configuration files, pass configuration information to other modules, and also supports users in editing and saving configuration information through the interface.

[0034] The database connection module 1021 is responsible for establishing and managing connections with the source and target databases. Based on configuration information, it uses appropriate database drivers or connection protocols to ensure the data migration tool can communicate with different database systems. This module also handles authentication and authorization issues during the connection process, ensuring the security and legitimacy of the data migration. The external file import module 1022 is used to import data from external files such as Excel, TXT, CSV, and JSON. During the data migration process, if the data source is not limited to databases but also includes data in various file formats, this module can read and import the data from these external files into the data migration tool, providing support for subsequent data conversion and loading. The data conversion module 1023 is used to perform conversion operations on data extracted from the source database or external files. Since different database systems or data storage formats may differ, the data conversion module can perform operations such as data cleaning, format conversion, and data type mapping according to predefined rules to ensure that the data can be correctly migrated to the target database and meet the structural and constraint requirements of the target database. The data extraction module 1024 is used to extract data from the source database. It reads the data from the source database according to the configured query statement or extraction rules to prepare for subsequent data conversion and loading. The data output module 1025 is used to load the converted data into the target database.

[0035] The data migration method provided in the embodiments of this application will now be described in conjunction with the accompanying drawings.

[0036] Figure 2 A flowchart illustrating a data migration method provided in one embodiment of this application is shown. Figure 2 As shown, the method may include the following steps: S201 to S204.

[0037] S201, Receive user input information, including configuration files, which include connection information between the source database and the target database, as well as data mapping rules.

[0038] The user input information consists of various instructions and data provided by the user for data migration; the configuration file is a file containing the connection information of the source database and the target database, as well as the data mapping rules, and is the core basis for data migration; the source database is the starting database for data migration, storing the data to be migrated; the target database is the ending database for data migration, used to receive the migrated data; and the data mapping rules are the mapping relationships between the data format and type of the source database and the data format and type of the target database.

[0039] In some embodiments, the configuration file allows users to flexibly define connection information between the source and target databases, data type mapping rules, tables to be migrated, and more. The configuration file also supports adding new database types and data type mapping rules, enabling support for more database systems. Users can extend the configuration file as needed to add new conversion rules, expand its applicability, and achieve data migration between different databases. Furthermore, separating configuration information from code simplifies the tool's code structure, eliminating the need for code modifications, offering high flexibility, and simplifying user operations.

[0040] In some embodiments, the configuration file may include database connection information, batch size, number of parallel threads, log file path, etc. Users can adjust these parameters according to actual needs to optimize data migration performance, improve tool efficiency and reliability, and it is highly configurable. The batch size (batch_size) defines the amount of data processed in each batch during the data migration process. Setting an appropriate batch size can improve data migration efficiency. The number of parallel threads (num_threads) defines the number of parallel threads used during data migration. Multi-threaded parallel processing can accelerate data migration.

[0041] In some embodiments, the configuration file may also include a log file path for recording operation logs and error logs, which helps in troubleshooting and performance monitoring, and improves the reliability and maintainability of the tool.

[0042] In some embodiments, the configuration file may also include external file configuration for importing data from external files into the target database. The tool reads the data from the external file according to the path and format in the configuration file and imports it into the corresponding database table. The external file configuration may include the external file type (e.g., Excel, TXT, JSON), the external file path (path), the worksheet name (sheet) in the Excel file, the delimiter (delimiter) in the TXT file, and the corresponding database table name (table), etc.

[0043] By receiving user input, the system allows users to customize data migration tasks according to their needs, providing a flexible configuration method. Centralized management of connection information between the source and target databases, as well as data mapping rules, through configuration files facilitates user modification and maintenance, improving the system's configurability and ease of use.

[0044] S202, Based on the connection information between the source database and the target database, establish communication connections with the source database and the target database respectively.

[0045] The connection information consists of specific parameters used to establish a communication connection with the database. These parameters may include the source and target database types (type), host address (host), port number (port), username (user), password (password), and database name (database), and are used by the migration system to connect to these two databases during runtime.

[0046] In some embodiments, the database type may include Wutong Database, the open-source relational database MySQL, the free software object-relational database management system PostgreSQL, and Oracle database, etc.

[0047] By utilizing the connection information between the source and target databases, connections can be accurately established with the corresponding databases, providing a stable communication foundation for subsequent data acquisition and transmission, and ensuring the smooth progress of data migration.

[0048] In one example, establishing a connection to a database can include: using a YAML library to read database connection information from a configuration file; using Python's database ORM and SQL utility library, SQLAlchemy, to create a database connection based on the connection information in the configuration file; verifying the validity of the connection information by attempting to connect to the database, and providing detailed error information when the connection fails; and using SQLAlchemy's connection pooling feature to manage database connections, automatically handling connection creation, reuse, and release, thereby improving performance and resource utilization.

[0049] S203, having obtained the migration data from the source database, the data type and format of the migration data are converted to the data type and format of the target database according to the data mapping rules to obtain the target migration data.

[0050] Among them, the migration data is the data that needs to be migrated from the source database, and the data format and type of the migration data are the same as those of the data in the source database; the target migration data is the data that has been transformed to meet the storage requirements of the target database; the data type is the classification of the data, such as integers, strings and dates, and different databases may define different type names; the data format is the storage form of the data.

[0051] Different database systems may support different data types, and even data types with the same name may be implemented differently in different databases. Therefore, it is necessary to map the data types of the source database to the corresponding data types of the target database.

[0052] The migration data from the source database is transformed according to the data mapping rules, so that the data can be adapted to the requirements of the target database, ensuring the integrity and consistency of the data during the migration process and avoiding data incompatibility issues.

[0053] S204, Send target migration data to the target database.

[0054] It realizes the actual migration of data from the source database to the target database, completes the transmission and storage of data between different databases, and enables the target database to obtain usable data.

[0055] This data migration method receives user input information for customized configuration, accurately establishes communication connections, and transforms data according to mapping rules to ultimately achieve data migration. It has high flexibility, configurability, and reliability, and can meet the needs of different users to migrate data between different databases.

[0056] In some embodiments, after establishing communication connections with the source database and the target database respectively based on the connection information between the source database and the target database, the method may further include:

[0057] Verify the validity of the database connection information to ensure a successful connection to the database;

[0058] In the event of a connection failure, provide detailed error information to help users troubleshoot the problem.

[0059] In some embodiments, such as Figure 3 As shown, this data migration method is used for client devices and server devices, and the method may include: S301 to S309.

[0060] S301, the client device receives the configuration file information input by the user and sends the configuration file information to the server device;

[0061] S302, after receiving the configuration file information, the server device establishes connections with the source database and the target database by parsing the configuration file information, and obtains the table information of the source database;

[0062] S303, The server device sends table information from the source database to the client device;

[0063] S304, The client device receives the table information of the source data sent by the source database and receives the migration table information selected by the user;

[0064] S305, the client device sends migration table information to the server device;

[0065] S306, The server device performs data migration to the source database based on the migration table information;

[0066] S307, The server device sends the migration task status to the client device;

[0067] S308, the client device receives the migration task status sent by the server device and displays it on the client device;

[0068] S309, The client device sends a control command for the migration task to the server device;

[0069] S309, the server device receives control commands from the client device and executes the corresponding operations.

[0070] The client and server devices work together, allowing users to complete complex data migration tasks with simple operations on the client device. The interface is intuitive, the operation is convenient, and the task status is reported in real time, improving the user experience.

[0071] In some embodiments, establishing communication connections with both the source and target databases based on the connection information between the source and target databases may include:

[0072] Based on the connection information between the source database and the target database, communication connections with the source database and the target database are established respectively through the abstraction layer device.

[0073] The abstraction layer device is an abstraction layer module that encapsulates the underlying communication details of different databases and provides a unified interface, thereby shielding the differences in the underlying protocols of different databases. It can load the corresponding driver to realize unified connection management of the database, enabling upper-layer applications to interact with various different databases in a unified way, so that migration tools do not need to directly handle the differences between different databases.

[0074] Using abstraction layer devices to establish communication connections simplifies the adaptation process to different databases. When dealing with various types of databases, it reduces development and maintenance costs, improves code reusability and portability, and enhances the flexibility and versatility of data migration.

[0075] In some embodiments, the connection information includes a communication protocol. Based on the connection information between the source database and the target database, communication connections with the source database and the target database are established respectively through an abstraction layer, which may include:

[0076] Based on the communication protocols of the source database and the target database, the corresponding communication protocol drivers for the source database and the target database are loaded through the abstraction layer device, and communication connections with the source database and the target database are established.

[0077] Among them, the communication protocol specifies a series of rules and standards that data should follow during the communication process. Different databases may use different communication protocols. The communication protocol is a driver program used to support a specific communication protocol. By loading the corresponding driver, the system can communicate with the database in accordance with the communication protocol.

[0078] By establishing a connection based on the communication protocols of the source and target databases and loading the corresponding communication protocol drivers, flexible adaptation to different databases can be achieved, improving system compatibility. Different databases may have different communication protocols; this approach ensures that the system can communicate effectively with various databases, guaranteeing broad applicability for data migration.

[0079] This method can load the corresponding driver according to the communication protocol, thereby achieving compatibility with multiple databases, expanding the applicability of the data migration method, and enabling it to meet the needs of different users to migrate data in different database environments.

[0080] In some embodiments, before obtaining migration data from the source database, the process may further include:

[0081] Obtain multiple data pointers to the migration data; each data pointer indicates the data location where the migration data is stored.

[0082] Migration data is retrieved in batches from the data locations corresponding to the data pointers.

[0083] A data pointer is a piece of information that indicates the location of data storage. It can be a cursor and is applicable to all relational databases that support cursors. Data pointers can be used to quickly locate the specific location of the data to be migrated in the source database.

[0084] By obtaining data pointers to indicate the storage location of the migrated data, the data that needs to be migrated can be quickly located, improving data retrieval efficiency. Batch data extraction is to avoid memory overflow issues caused by reading large amounts of data at once. Data is extracted from the source database in batches according to the batch size defined in the configuration file, reducing the number of database operations and improving extraction efficiency.

[0085] The method of acquiring data in batches, combined with the positioning function of data pointers, optimizes the data acquisition process, improves the efficiency and reliability of data migration, reduces the consumption of system resources, and is suitable for large-scale data migration scenarios.

[0086] In some embodiments, retrieving migration data in batches from the data locations of the migration data corresponding to the data pointers may further include:

[0087] Add batch and row numbers to the extracted data to ensure data order and integrity.

[0088] In some embodiments, the configuration process for data mapping rules may include:

[0089] Determine the source and target database types that need to be supported in the project, such as WutongDB, MySQL, PostgreSQL, Oracle, SQL Server, Vertica, etc.

[0090] Obtain the official documentation for each database system, which lists the supported data types and their formats, such as length, precision, and default values. The official documentation can be obtained by visiting the official website or developer documentation center of each database system and downloading or browsing the relevant data type support documentation.

[0091] Compare the representation and characteristics of the same or similar data types in each database, and record the specific implementation differences of each data type in different database systems, such as length limits and precision differences. In the comparison process, pay attention to the specific implementation differences of each data type, such as length limits, whether indexes are supported, and whether null values ​​are supported. Record the differences of each data type in a table or document to facilitate the subsequent formulation of mapping rules.

[0092] Based on the comparison results, define the mapping relationship between the source database type and the target database type, and consider the length and precision of the fields in the mapping rules to ensure that the converted data types can store data correctly.

[0093] In some embodiments, after configuring the data mapping rules, the method may further include:

[0094] Select representative sample data for testing to ensure that the mapping rules can correctly convert data types;

[0095] Check whether the converted data maintains the integrity and consistency of the source data.

[0096] By testing and verifying the data mapping rules, it can be ensured that the mapping rules can correctly convert data types.

[0097] After the testing is completed, user feedback can be collected from actual use. Based on the feedback and new requirements, the data type mapping rules in the configuration file can be updated to ensure that the tool can cope with the ever-changing database environment and requirements.

[0098] In some embodiments, the data format includes data field length and precision, table structure, and data format; the data mapping rules include mapping rules for data field length and precision, mapping rules for table structure, and mapping rules for data format, such as... Figure 4 As shown, the data type and data format of the migration data are converted into the data type and data format of the target database according to the data mapping rules to obtain the target migration data, which may include: S401 to S404.

[0099] S401, Map the data type of the migrated data to the data type of the target database according to the data rules;

[0100] S402, Map the data field length and precision of the migrated data to the data field length and precision of the target database according to the mapping rules for data field length and precision;

[0101] S403, Map the table structure of the migrated data to the table structure of the target database according to the table structure mapping rules;

[0102] S404, mapping the data format of the migrated data to the mapping rule of the target data according to the data format mapping rule.

[0103] Among them, the length and precision of the data field refer to the allowed length of the data during storage and the precision requirements such as the number of decimal places; the table structure describes the composition of the tables in the database, including information such as table name, column name, data type, default value, and constraints; the data format specifies the way the data is stored and represented, such as date format and string encoding.

[0104] Furthermore, data type mapping is responsible for determining the correspondence between data types; field length and precision adjustment adapts fields according to the requirements of the target database; table structure conversion converts the table structure information of the source database into the table structure definition of the target database; and data format conversion ensures that the data is stored in the correct format in the target database.

[0105] These interconnected parts work together to complete the data structure conversion, ensuring accurate and efficient data migration between different database systems. Data type mapping converts the data types of the source database to the corresponding data types of the target database, which is the first step in data structure conversion. Subsequent steps are based on the results of data type mapping. Field length and precision adjustment is performed based on the results of data type mapping; the adjusted field length and precision information will be used to generate the table structure of the target database, ensuring that the data can be correctly stored in the target database. Table structure conversion is performed based on the results of field length and precision adjustment, converting the table structure information of the source database to the corresponding table structure of the target database. The converted table structure information will be used for data format conversion. Data format conversion is performed based on the results of table structure conversion, ensuring that the data is stored correctly in the target database. The converted data format information will be used for final data storage. In this way, data structure conversion between databases is achieved, ensuring the consistency and integrity of data during the migration process.

[0106] By performing comprehensive data transformation through detailed data mapping rules, the compatibility issues between different databases can be effectively resolved, improving the quality and reliability of data migration and ensuring the availability and consistency of data after migration.

[0107] In some embodiments, sending target migration data to the target database may include:

[0108] Generate a data migration task based on the target migration data;

[0109] The data migration task is broken down into multiple migration subtasks, and the target migration data corresponding to the migration subtasks is sent to the target database in parallel based on the configured number of parallel threads.

[0110] Among them, the data migration task is a task unit that encapsulates the data to be migrated and its related information; the migration subtask is the result of decomposing the data migration task into multiple smaller subtasks, each subtask corresponding to the migration of a part of the data; the number of parallel threads is the number of threads that execute the migration subtasks at the same time, used to control resource consumption and migration speed.

[0111] By breaking down the data migration task into multiple subtasks and sending data to the target database in parallel based on the configured number of parallel threads, the system's resources can be fully utilized, the speed and efficiency of data transmission can be improved, the data migration time can be shortened, and the system's performance can be enhanced.

[0112] Existing tools may be inefficient when processing large-scale data. For example, pg_dump and pg_restore are slow when exporting and importing large amounts of data, especially when data type conversion is required. The embodiments of this application improve data migration efficiency through task decomposition and parallel sending mechanisms, fully leveraging the advantages of multithreading. This makes them suitable for large-scale data migration scenarios and meets users' needs for rapid data migration.

[0113] In some embodiments, the data migration task can be divided into multiple migration subtasks according to the batch size.

[0114] In one example, sending target migration data to the target database may include:

[0115] The LOAD DATA LOCAL INFILE command in MySQL writes data to a memory file and then inserts data from the target database's memory using the same command.

[0116] The COPY command in PostgreSQL databases uses the copy_expert method of the psycopg2 library to insert data in batches from the target database's memory.

[0117] The batch insert command in Oracle database uses the executemany method of the cx_Oracle library to implement batch data insertion.

[0118] The batch insert command for SQL Server databases uses the executemany method of the pyodbc library to implement batch data insertion.

[0119] In some embodiments, sending target migration data corresponding to migration subtasks to the target database in parallel based on the configured number of parallel threads may include:

[0120] During the parallel transmission of target migration data corresponding to migration subtasks to the target database, the transmission status of each migration subtask is monitored.

[0121] If the transmission status of the target migration subtask is detected to be the target status, stop sending the target migration data corresponding to the target migration subtask to the target database, and resend the target migration data corresponding to the migration subtask to the target database. The target migration subtask can be any migration subtask.

[0122] The transmission status refers to the real-time status of the migration subtask, which may include states such as in progress, success, and failure; the target status refers to the abnormal state that needs to be triggered for retry, which may include states such as network interruption and target database write failure.

[0123] During parallel data transmission, monitoring the transmission status of each subtask allows for timely detection of problems such as transmission failures. When the transmission status of the target migration subtask is detected to be in the target state, the transmission of the corresponding data for that subtask is stopped and retransmitted, ensuring data integrity and correctness and guaranteeing successful data migration to the target database.

[0124] By monitoring the transmission status and resending failed subtasks, the reliability and stability of data migration are improved, the risk of data loss or errors is reduced, and the quality of data migration is ensured.

[0125] In some embodiments, receiving user input information may include:

[0126] Receives user input information through the user interface.

[0127] The user interface is the interface through which users interact with the system. Users can input necessary information, such as configuration files, through the interface.

[0128] It provides users with a user-friendly interaction method, enabling non-technical personnel to easily perform data migration operations, lowering the barrier to entry, improving user experience, and enhancing system availability.

[0129] By using a user interface to receive input information, the user operation process is simplified, the system's usability is improved, and more users can easily use this data migration method.

[0130] In some embodiments, the user interface may include interfaces such as a graphical user interface and a command-line interface.

[0131] In some embodiments, the graphical user interface (GUI) provides an intuitive and easy-to-use interface that allows users to easily configure, start, and monitor data migration tasks. Within the GUI, users can select and load configuration files, edit their contents (such as database connection information and data type mapping rules) using interface elements, and save the modified configuration files. Users can control the data migration task, starting, pausing, resuming, and terminating it. The GUI also allows for status monitoring of the data migration task, displaying real-time progress, warnings and error messages that occur during the migration process, and detailed operation logs.

[0132] In some embodiments, the command-line interface provides a concise and efficient command-line operation interface, allowing users to modify configuration file contents through command-line parameters. This enables users to configure, start, and monitor data migration tasks via the command line. Data migration tasks can be started, paused, resumed, and stopped via the command line; the command line displays the data migration progress, warnings and error messages that occur during the migration process, and detailed operation logs.

[0133] In some embodiments, the migration data in the source database can be migration data from external files. Obtaining migration data from external files may include:

[0134] Read the external file path and data extraction parameters from the configuration file, such as batch size and file type;

[0135] Select the corresponding parsing method based on the file type, and extract data in batches according to the batch size defined in the configuration file;

[0136] The extracted data is preprocessed, such as removing null values ​​and converting formats, to obtain the migration data of the external file.

[0137] In some embodiments, the external file format may include multiple file formats such as Excel, TXT, CSV, and JSON.

[0138] In some embodiments, after establishing communication connections with the source database and the target database respectively, the method may further include:

[0139] The connection relationship between the source database and the target database is controlled through the connection pool device.

[0140] The connection pool device is used to manage database connections. It can pre-create a certain number of database connections and allocate existing connections to applications for reuse when needed, and return the connections to the connection pool after use.

[0141] While a database connection typically requires only a single setup, large-scale data migrations may involve multiple database operations, such as data extraction, transformation, and insertion. These operations may require frequent database accesses. Using a connection pool can reduce the overhead of establishing and releasing connections for each operation. Connection pooling devices manage connections between the source and target databases automatically, handling connection creation, reuse, and release. This reduces the number of connection creations and destructions, improving system performance and resource utilization. Furthermore, connection pooling devices provide unified management and monitoring of connections, facilitating connection maintenance and optimization, and enhancing system stability and reliability.

[0142] Furthermore, in concurrent processing scenarios, such as multi-threaded and multi-process environments, multiple database connections are required for simultaneous data migration operations. Connection pools can effectively manage these concurrent connections, improving the performance and efficiency of concurrent processing. Connection pools can control the maximum number of connections, preventing excessive database connections from overloading the database server. They can also manage idle connections, preventing connections from wasting resources by remaining idle for extended periods.

[0143] The use of connection pool devices optimizes database connection management, improves system performance and resource utilization, enhances system stability and maintainability, and facilitates long-term stable data migration tasks.

[0144] In some embodiments, controlling the connection relationship between the source database and the target database through a connection pool device may include:

[0145] If the duration of a target connection to the source database or target database exceeds a set threshold, the connection to the database whose duration of the target connection exceeds the set threshold will be disconnected via the connection pool device.

[0146] The target connection duration is the duration after a connection to the database is established; the threshold is a pre-defined time value, and when the connection duration exceeds this value, the database connection is automatically disconnected to release resources.

[0147] For long-running data migration tasks, connection pools can automatically manage connection timeouts and failures, ensuring connection stability and reliability during extended operation. When the connection duration to the source or target database exceeds a set threshold, the connection pool device disconnects the corresponding database connection. This prevents connections from occupying system resources for extended periods without being effectively utilized, optimizes connection resource allocation, improves overall system performance and efficiency, and avoids system problems caused by excessive or prolonged connection unresolved connections.

[0148] Mechanisms that monitor connection duration and disconnect timed-out connections can effectively manage connection resources, prevent resource waste, and improve system performance and stability.

[0149] In some embodiments, the target connection duration is the duration of a spatial connection that has not been used for database operations after a connection has been established with the database.

[0150] In some embodiments, the method may further include:

[0151] The logging device records various operations and events that occur during the data migration process, including successful operations, errors, and warnings.

[0152] In some embodiments, logging various operations and events that occur during the data migration process using a logging device may include:

[0153] Read the parameters for logging from the configuration file, such as log level and log output method;

[0154] Configure the logger according to the parameters in the configuration file, including the log format, log level, and output method;

[0155] Use a logger to record operation logs, error logs, and warning logs for each step of the data migration process;

[0156] Output the recorded log information to a configured target, which may include the console or a file.

[0157] It can comprehensively record all key operations and events during the data migration process, ensuring the integrity of log information. It can record log information in real time, ensuring that relevant information can be obtained promptly in the event of errors. Furthermore, it supports multiple log output methods and log levels, which users can configure and adjust as needed.

[0158] In some embodiments, the method may further include:

[0159] In the event of an error message, the error processing device handles the error message, thereby enabling centralized management of error messages during the migration process.

[0160] In some embodiments, error messages may include database connection errors, configuration file errors, data extraction errors, data conversion errors, and data migration errors.

[0161] A database connection error occurs when attempting to connect to the database during initialization, such as connection failure or timeout. The error log is logged, the administrator is notified, and a reconnection attempt is made. If the retry is successful, subsequent operations continue; otherwise, the current task is terminated and the failure information is logged.

[0162] A configuration file error occurs when loading a configuration file, such as a missing file or an incorrect format. The error is logged, the administrator is notified, the user is prompted to check the configuration file, and system initialization is stopped, awaiting the user's correction of the configuration file.

[0163] Data extraction errors occur when retrieving data from the source database, such as query execution failure or data reading errors. Log the error, notify the administrator, and attempt to re-execute the data extraction operation. If the retry is successful, continue with subsequent operations; otherwise, skip the current task and log the failure information.

[0164] Data conversion errors occur during the data conversion process, such as data format conversion errors or data validation failures. An error log is logged, the administrator is notified, and the data conversion operation is retried. If the retry is successful, subsequent operations continue; otherwise, the data that cannot be converted is recorded in the error log or backup file.

[0165] Data migration errors occur when data is inserted into the target database, such as insertion failure or data constraint conflicts. An error log is logged, the administrator is notified, and the data insertion operation is retried. If the retry is successful, subsequent operations continue; otherwise, the data that could not be inserted is recorded in the error log or backup file.

[0166] Centralized management of error handling logic ensures consistency between log recording and notifications. All error logs and notifications are processed through a unified error handling device. When an error occurs, the corresponding modules at each step first transmit the error information to the error handling device for centralized recording and notification, and then execute the customized error handling logic to ensure that all errors are correctly recorded and promptly notified to relevant personnel.

[0167] Error handling logic allows each module to customize its error handling logic according to specific needs, such as retry mechanisms and recovery operations. Each module can independently optimize and adjust its error handling logic without affecting other modules. Furthermore, existing tools typically have limited error handling and recovery capabilities during migration. For example, SQL Server's SSIS often requires manual intervention when encountering errors during complex data migration tasks. This application's embodiments, however, design a robust error handling mechanism that can automatically detect and recover from anomalies during the migration process, ensuring the stability and integrity of the data migration.

[0168] Figure 5 This application illustrates a data migration apparatus 500 provided in an embodiment of the present application. The apparatus may include:

[0169] The receiving module 501 is used to receive user input information, the input information including a configuration file, the configuration file including connection information between the source database and the target database, and data mapping rules;

[0170] The connection module 502 is used to establish communication connections with the source database and the target database respectively based on the connection information between the source database and the target database;

[0171] The conversion module 503 is used to convert the data type and data format of the migration data into the data type and data format of the target database according to the data mapping rules when the migration data of the source database is obtained, so as to obtain the target migration data.

[0172] The sending module 504 is used to send the target migration data to the target database.

[0173] In some embodiments, the connection module 502 is further configured to establish communication connections with the source database and the target database respectively through the abstraction layer device based on the connection information between the source database and the target database.

[0174] In some embodiments, the connection module 502 is further configured to load the communication protocol drivers corresponding to the source database and the target database respectively through the abstraction layer device according to the communication protocol of the source database and the target database, and establish a communication connection with the source database and the target database.

[0175] In some embodiments, the data migration apparatus 500 may further include:

[0176] The acquisition module is used to acquire multiple data pointers to the migration data, which indicate the data location where the migration data is stored.

[0177] The acquisition module is also used to acquire migration data in batches from the data locations of the migration data corresponding to the data pointers.

[0178] In some embodiments, the data migration apparatus 500 may further include:

[0179] The mapping module is used to map the data type of the migrated data to the data type of the target database according to data rules;

[0180] The mapping module is also used to map the data field length and precision of the migrated data to the data field length and precision of the target database according to the mapping rules for data field length and precision;

[0181] The mapping module is also used to map the table structure of the migrated data to the table structure of the target database according to the mapping rules of the table structure;

[0182] The mapping module is also used to map the data format of the migrated data to the mapping rules of the target data according to the data format mapping rules.

[0183] In some embodiments, the data migration apparatus 500 may further include:

[0184] The generation module is used to generate data migration tasks based on the target migration data;

[0185] The sending module 504 is also used to decompose the data migration task into multiple migration subtasks, and send the target migration data corresponding to the migration subtasks to the target database in parallel based on the configured number of parallel threads.

[0186] In some embodiments, the data migration apparatus 500 may further include:

[0187] The detection module is used to detect the transmission status of each migration subtask during the process of sending the target migration data corresponding to the migration subtask to the target database in parallel.

[0188] The sending module 504 is further configured to, when detecting that the transmission status of the target migration subtask is the target status, stop sending the target migration data corresponding to the target migration subtask to the target database, and resend the target migration data corresponding to the migration subtask to the target database, wherein the target migration subtask is any migration subtask.

[0189] In some embodiments, the receiving module 501 is further configured to receive input information from the user through a user interface.

[0190] In some embodiments, the connection module 502 is further configured to control the connection relationship between the source database and the target database through the connection pool device.

[0191] In some embodiments, the connection module 502 is further configured to disconnect the connection with the database whose target connection duration exceeds the set threshold by controlling the connection pool device if the target connection duration with the source database or the target database exceeds the set threshold.

[0192] Figure 5 The various modules in the illustrated device can achieve Figure 1 The various steps involved, and the corresponding technical effects achieved, will not be elaborated upon here for the sake of brevity.

[0193] Figure 6 A schematic diagram of the hardware structure of the terminal device provided in an embodiment of this application is shown.

[0194] The terminal device may include a processor 601 and a memory 602 storing computer program instructions.

[0195] Specifically, the processor 601 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0196] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be non-volatile solid-state memory. Memory 602 may be internal or external to the integrated gateway disaster recovery device.

[0197] In one example, memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the data migration method according to this disclosure.

[0198] The processor 601 reads and executes computer program instructions stored in the memory 602 to achieve... Figure 1 The data migration method in the illustrated embodiment.

[0199] In one example, the terminal device may further include a communication interface 603 and a bus 604. Wherein, for example... Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 604 and complete communication with each other.

[0200] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0201] Bus 604 includes hardware, software, or both, that couples components of an end device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0202] Furthermore, in conjunction with the data migration methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data migration methods described in the above embodiments.

[0203] This application also provides a computer program product, including a computer program, which, when executed, implements any of the data migration methods described in the above embodiments.

[0204] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0205] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or text segments used to perform the required tasks. Programs or text segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Text segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0206] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0207] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0208] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data migration method, characterized by, The method comprises: receiving input information of a user, the input information comprising a configuration file, the configuration file comprising connection information of a source database and a target database, and a data mapping rule; based on the connection information of the source database and the target database, establishing a communication connection with the source database and the target database respectively; in the case of obtaining migration data of the source database, converting the data type and data format of the migration data into the data type and data format of the target database according to the data mapping rule to obtain target migration data; sending the target migration data to the target database.

2. The data migration method of claim 1, wherein, The method comprises: based on the connection information of the source database and the target database, establishing a communication connection with the source database and the target database respectively through an abstraction layer device.

3. The data migration method of claim 2, wherein, The connection information comprises a communication protocol, and the method comprises: based on the connection information of the source database and the target database, establishing a communication connection with the source database and the target database respectively through an abstraction layer.

4. The data migration method of claim 1, wherein, According to the communication protocols of the source database and the target database, the corresponding communication protocol drivers of the source database and the target database are loaded through an abstraction layer device to establish a communication connection with the source database and the target database. Before obtaining the migration data of the source database, the method further comprises: obtaining a plurality of data pointers of the migration data, the data pointers indicating the data positions where the migration data is stored; 5. The data migration method of claim 1, wherein, batching the migration data from the data positions corresponding to the data pointers. The data format comprises data field length and precision, table structure and data format, and the data mapping rule comprises a mapping rule of data type, a mapping rule of data field length and precision, a mapping rule of table structure and a mapping rule of data format, and the method comprises: mapping the data type of the migration data into the data type of the target database according to the mapping rule of the data type; mapping the data field length and precision of the migration data into the data field length and precision of the target database according to the mapping rule of the data field length and precision; mapping the table structure of the migration data into the table structure of the target database according to the mapping rule of the table structure; 6. The data migration method of claim 1, wherein, mapping the data format of the migration data into the mapping rule of the target data according to the mapping rule of the data format. The method comprises: generating a data migration task according to the target migration data; 7. The data migration method of claim 6, wherein, decomposing the data migration task into a plurality of migration subtasks, and sending the target migration data corresponding to the migration subtasks to the target database in parallel based on the configured number of parallel threads. The method comprises: In the process of sending the target migration data corresponding to the migration subtask to the target database in parallel, the transmission state of each migration subtask is detected; In the case where the transmission state of the target migration subtask is detected as the target state, the sending of the target migration data corresponding to the target migration subtask to the target database is stopped, and the target migration data corresponding to the migration subtask is sent to the target database again, the target migration subtask being any migration subtask.

8. The data migration method of claim 1, wherein, The input information of the user includes: The input information of the user through the user interface is received.

9. The data migration method of claim 1, wherein, After the communication connections with the source database and the target database are established respectively, the method further includes: The connection relationship between the source database and the target database is controlled through the connection pool device.

10. The data migration method of claim 9, wherein, The connection relationship between the source database and the target database is controlled through the connection pool device, including: In the case where the target connection time length with the source database or the target database exceeds the set threshold, the connection with the database whose target connection time length exceeds the set threshold is controlled to be disconnected through the connection pool device.

11. A data migration apparatus, characterized by comprising: The device includes: The receiving module is configured to receive the input information of the user, the input information including a configuration file, the configuration file including connection information of the source database and the target database, and a data mapping rule; The connection module is configured to establish communication connections with the source database and the target database based on the connection information of the source database and the target database; The conversion module is configured to, in the case where the migration data of the source database is obtained, convert the data type and the data format of the migration data into the data type and the data format of the target database according to the data mapping rule, to obtain target migration data; The sending module is configured to send the target migration data to the target database.

12. A terminal device, comprising: The device includes a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the data migration method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the data migration method of any one of claims 1-10.

14. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the electronic device, so that the electronic device executes the data migration method of any one of claims 1-10.

Images