Data migration method and device, terminal equipment and computer readable storage medium

Abstract

The application discloses a data migration method and device, terminal equipment and a computer readable storage medium, relates to the technical field of data processing, and the data migration method comprises the following steps: determining database connection information between a source database and a target database, and creating keyword mapping information based on the database connection information; creating an initial data migration task flow batch based on the database connection information, and processing the initial data migration task flow batch to obtain a target data migration task flow batch; obtaining target migration data through the target data migration task flow batch and the keyword mapping information, and loading the target migration data to the target database. The data migration method can achieve the technical effect that the terminal equipment can migrate data between databases with different data structures.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data migration method, apparatus, terminal device, and computer-readable storage medium. Background Technology

[0002] Currently, when technicians need to maintain or prepare a database to resolve data anomaly issues, they must first migrate the data in the database to be maintained to another database before they can perform maintenance or preparation on the database.

[0003] In related technologies, technicians can only choose to use ETL (Extract-Transform-Load, data warehouse technology) tools to perform extraction, transformation and loading operations to migrate data from the database to be maintained to the target database.

[0004] However, the data structures of different databases vary greatly, and ETL tools can only perform data migration operations between databases with the same data structure. Therefore, it is difficult for technicians to use ETL tools to migrate data between databases with different data structures. Summary of the Invention

[0005] The main objective of this application is to provide a data migration method, apparatus, terminal device, and computer-readable storage medium, which enables the terminal device to migrate data between databases with different data structures.

[0006] To achieve the above objectives, this application provides a data migration method, which includes the following steps:

[0007] Determine the database connection information between the source database and the target database, and create keyword mapping information based on the database connection information;

[0008] An initial data migration task flow batch is created based on the database connection information, and the initial data migration task flow batch is processed to obtain the target data migration task flow batch.

[0009] The target migration data is obtained by using the target data migration task flow batch and the keyword mapping information, and then the target migration data is loaded into the target database.

[0010] Furthermore, the step of determining the database connection information between the source database and the target database includes:

[0011] Receive data source configuration information;

[0012] The database connection information between the source database and the target database is determined based on the data source configuration information.

[0013] Furthermore, the step of creating keyword mapping information based on the database connection information includes:

[0014] Based on the database connection information, determine the first data type corresponding to the source database and the second data type corresponding to the target database;

[0015] Based on the first data type and the second data type, determine the target keyword corresponding to the data to be migrated from the source database in the target database;

[0016] Keyword mapping information is created based on the target keywords.

[0017] Further, the step of processing the initial data migration task flow batch to obtain the target data migration task flow batch includes:

[0018] Identify multiple data migration tasks within the initial data migration task flow batch, and generate a target task flow based on the multiple data migration tasks;

[0019] The target data migration task flow batch is obtained by arranging the multiple data migration tasks according to the target task flow.

[0020] Further, the step of obtaining the target migration data through the target data migration task flow batch and the keyword mapping information includes:

[0021] The data type of the data to be migrated in the source database is modified by the target data migration task flow batch and the keyword mapping information to obtain standard file data, wherein the data type of the standard file data is a third data type;

[0022] Based on the keyword mapping information, the data type of the standard file data is modified from the third data type to the second data type to obtain the target migration data.

[0023] Furthermore, after the step of creating an initial data migration task flow batch based on the database connection information, the method further includes:

[0024] Determine whether the initial data migration task batch contains large-volume tasks;

[0025] If it is determined that the initial data migration task flow batch contains the large-volume task, then a preset splitting rule is obtained, and the large-volume task is split into multiple small-volume tasks according to the splitting rule.

[0026] Multiple small-volume tasks are added to the initial data migration task flow batch to obtain the target data migration task flow batch.

[0027] Furthermore, the step of determining whether the initial data migration task stream batch contains large-volume tasks includes:

[0028] The initial data migration task flow batch is inspected to obtain the migration data table corresponding to each of the multiple data migration tasks within the initial data migration task flow batch.

[0029] Determine the amount of migration data corresponding to each of the migration data tables, and obtain a preset data volume threshold;

[0030] The migration data volume of each data point is compared with the data volume threshold to obtain multiple comparison results;

[0031] When multiple comparison results include a target comparison result where the amount of migrated data is greater than the data volume threshold, it is determined that the initial data migration task flow batch contains a large-volume task.

[0032] Furthermore, to achieve the above objectives, this application also provides a data migration apparatus, the apparatus comprising:

[0033] The configuration management module is used to determine the database connection information between the source database and the target database, and to create keyword mapping information based on the database connection information;

[0034] The task creation module is used to create an initial data migration task flow batch based on the database connection information, and to process the initial data migration task flow batch to obtain the target data migration task flow batch.

[0035] The task execution module is used to obtain target migration data through the target data migration task flow batch and the keyword mapping information, and load the target migration data into the target database.

[0036] In addition, to achieve the above objectives, this application also provides a terminal device, the terminal device comprising: a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the computer program, when executed by the processor, implements the steps of the data migration method described above.

[0037] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data migration method described above.

[0038] The data migration method, apparatus, terminal device, and computer-readable storage medium provided in this application embodiment determine the database connection information between the source database and the target database, and create keyword mapping information based on the database connection information; create an initial data migration task flow batch based on the database connection information, and process the initial data migration task flow batch to obtain a target data migration task flow batch; obtain target migration data through the target data migration task flow batch and the keyword mapping information, and load the target migration data into the target database.

[0039] In this embodiment, when a user needs to migrate data between databases with different data structures, the terminal device first determines the database connection information between the source database and the target database to be migrated, and then creates key mapping information between the source database and the target database based on the database connection information. After that, the terminal device creates an initial data migration task flow batch based on the database connection information, and arranges the initial data migration task flow batch to obtain a target data migration task flow batch. Finally, the terminal device executes the target data migration task flow batch, thereby obtaining the target migration data corresponding to the data structure of the target database through the target data migration task flow batch and the key mapping information, and loading the target migration data into the target database to complete the data migration operation.

[0040] Thus, this application determines the database connection information between databases with different data structures, creates key mapping information corresponding to each piece of data to be migrated within the database based on the database connection information, converts the data to be migrated to a standard data file based on the key mapping information, and modifies the standard data file to match the target migration data of the target database based on the key mapping information. This enables databases with different data structures to be connected, thereby solving the technical problem in related technologies where ETL tools cannot migrate data between databases with different data structures, and achieving the technical effect of enabling terminal devices to migrate data between databases with different data structures. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of the terminal device in the hardware operating environment involved in the embodiments of this application;

[0042] Figure 2 This is a flowchart illustrating the first embodiment of the data migration method of this application;

[0043] Figure 3 This is a schematic diagram of the migration process involved in one embodiment of the data migration method of this application;

[0044] Figure 4This is a schematic diagram of the task creation process involved in one embodiment of the data migration method of this application;

[0045] Figure 5 This is a schematic diagram of the task execution process involved in an embodiment of the data migration method of this application;

[0046] Figure 6 This is a schematic diagram of the server allocation process involved in one embodiment of the data migration method of this application;

[0047] Figure 7 This is a flowchart illustrating a preferred embodiment of the data migration method of this application;

[0048] Figure 8 This is a schematic diagram of the functional modules involved in an embodiment of the data migration device of this application.

[0049] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0051] Reference Figure 1 , Figure 1 This is a schematic diagram of the terminal device structure of the hardware operating environment involved in the embodiments of this application.

[0052] It should be noted that the terminal device in this application embodiment can be a device that executes the data migration method of this application. Specifically, the terminal device can be a mobile terminal, data storage control terminal, PC, or other terminal that is internally configured with a multi-source heterogeneous database end-to-end data migration system.

[0053] like Figure 1As shown, the terminal device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0054] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0055] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and computer programs.

[0056] exist Figure 1 In the terminal device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the terminal device of this application can be set in the terminal device, and the terminal device calls the computer program stored in the memory 1005 through the processor 1001 and executes the following embodiments.

[0057] Based on the aforementioned terminal equipment, the overall concept of the data migration method of this application is provided.

[0058] Currently, when technicians need to maintain or prepare a database to resolve data anomaly issues, they must first migrate the data in the database to be maintained to another database before they can perform maintenance or preparation on the database.

[0059] In related technologies, technicians can only choose to use ETL (Extract-Transform-Load, data warehouse technology) tools to perform extraction, transformation and loading operations to migrate data from the database to be maintained to the target database.

[0060] However, the data structures of different databases vary greatly, and ETL tools can only perform data migration operations between databases with the same data structure. Therefore, it is difficult for technicians to use ETL tools to migrate data between databases with different data structures.

[0061] To address the aforementioned issues, this application proposes a data migration method comprising the following steps: determining database connection information between a source database and a target database, and creating keyword mapping information based on the database connection information; creating an initial data migration task flow batch based on the database connection information, and processing the initial data migration task flow batch to obtain a target data migration task flow batch; obtaining target migration data through the target data migration task flow batch and the keyword mapping information, and loading the target migration data into the target database.

[0062] Thus, this application determines the database connection information between databases with different data structures, creates key mapping information corresponding to each piece of data to be migrated within the database based on the database connection information, converts the data to be migrated to a standard data file based on the key mapping information, and modifies the standard data file to match the target migration data of the target database based on the key mapping information. This enables databases with different data structures to be connected, thereby solving the technical problem in related technologies where ETL tools cannot migrate data between databases with different data structures, and achieving the technical effect of enabling terminal devices to migrate data between databases with different data structures.

[0063] Based on the above-described terminal device and the overall concept of the data migration method of this application, various embodiments of the data migration method of this application are further proposed.

[0064] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data migration method of this application.

[0065] It should be understood that although the logical order is shown in the flowchart, in some cases the data migration method of this application may of course perform the steps shown or described in a different order than that shown here.

[0066] like Figure 2 As shown, in this embodiment, the data migration method of this application may include the following steps:

[0067] Step S10: Determine the database connection information between the source database and the target database, and create keyword mapping information based on the database connection information;

[0068] Step S20: Create an initial data migration task flow batch based on the database connection information, and process the initial data migration task flow batch to obtain the target data migration task flow batch;

[0069] Step S30: Obtain target migration data through the target data migration task flow batch and the keyword mapping information, and load the target migration data into the target database;

[0070] It should be noted that the source database and the target database should be databases containing file data of different data types. The database is divided into two categories: internal database and external database. It can be understood that the internal database is the database whose configuration information has been recorded by the terminal device, while the external database is the database whose configuration information has not been recorded by the terminal device (i.e., the newly introduced database). The specific types of the database include, but are not limited to: TD (TeraDate), historical database, offline platform, exchange platform and Gauss cluster, etc.

[0071] In this embodiment, when a user needs to migrate data between databases with different data structures, the terminal device first determines the source database from which the data to be migrated needs to be extracted and the target database from which the data needs to be loaded. It then calls its configured data source management module to determine the database connection information between the source and target databases. Based on this connection information, the data source management module creates keyword mapping information between the source and target databases. Next, the data source management module inputs this database connection information and the keyword mapping information to the configuration initiation service module configured within the terminal device. The configuration initiation service module creates an initial data migration task flow batch based on the database connection information, and creates multiple data migration tasks under the initial data migration task flow batch. These multiple data migration tasks are then arranged to obtain a target data migration task flow batch. Finally, the data source management module inputs this target data migration task flow batch to the execution monitoring service module within the terminal device. The execution monitoring service module executes the target data migration task flow batch, thereby modifying the data type of the data to be migrated based on the keyword mapping information to obtain target migration data that matches the target database. This target migration data is then loaded into the target database to complete the data migration operation.

[0072] For example, when a user needs to migrate data between an in-warehouse database and an external database, the terminal device first identifies the in-warehouse database as the source database from which data needs to be extracted, and the external database as the target database from which data needs to be loaded. The terminal device then calls its configured data source management module to determine the database connection information between the source and target databases, and creates keyword mapping information based on this connection information. Afterward, the data source management module inputs this connection information and keyword mapping information into the terminal device's configured configuration initiation service module. The configuration initiation service module then uses this connection information and keyword mapping information to obtain the source database configuration information filled in by the user, including the source platform, source data source, source schema name, source table name, target platform, target data source, target schema name, target table name, and filter conditions. Based on this source database configuration information, it creates an initial data migration task. The service module initiates a batch of data migration tasks, creating multiple data migration tasks within the initial batch, including tasks for automatically generating the target database table, automatically generating data migration configurations, automatically generating task scheduling workflow configurations, automatically executing data migration tasks, and updating status. This process involves arranging these tasks to obtain the target data migration task batch. Finally, the service module inputs this target data migration task batch into the execution monitoring service module configured on the terminal device. This execution monitoring service module executes each data migration task within the target data migration task batch, modifying the data type of the data to be migrated from the first data type corresponding to the source database to the second data type matching the target database based on the keyword mapping information. The execution monitoring service module then loads this target migration data into the target database to complete the data migration operation.

[0073] Thus, this application determines the database connection information between databases with different data structures, creates key mapping information corresponding to each piece of data to be migrated within the database based on the database connection information, converts the data to be migrated to a standard data file based on the key mapping information, and modifies the standard data file to match the target migration data of the target database based on the key mapping information. This enables databases with different data structures to be connected, thereby solving the technical problem in related technologies where ETL tools cannot migrate data between databases with different data structures, and achieving the technical effect of enabling terminal devices to migrate data between databases with different data structures.

[0074] Furthermore, it should be noted that in this embodiment and another embodiment, when the source file path information filled in by the user in the configuration information corresponds to a standard data file that corresponds to a preset standard file specification, the terminal device does not need to determine the source database, but directly determines the target database selected by the user, and creates keyword mapping information based on the target database. Then, based on the keyword mapping information, the data type of the standard data file is modified to a second data type to obtain the target migration file, and the target migration file is input into the target database to complete the data migration operation. In this way, the user can directly load the standard file data into the target database without extracting the data to be migrated from the database to be migrated.

[0075] Furthermore, it should be noted that in this embodiment and another embodiment, after the execution monitoring service module obtains the target data migration task flow batch, it also needs to update the status of the target data migration task flow batch and generate a visualization interface based on the status. The execution monitoring service module then displays the visualization interface to the user through the display module configured on the terminal device itself. In this way, the user can view the execution status of the target data migration task flow batch based on the visualization interface and manually intervene in the target data migration task flow batch by rerunning, redeploying, canceling, setting, or forcibly stopping it.

[0076] In addition, please refer to this embodiment and another embodiment. Figure 3 , Figure 3 This is a schematic diagram of the migration process involved in one embodiment of the data migration method of this application, as shown below. Figure 3 As shown, when a user needs to migrate data between an in-warehouse database and an external database, the user can execute a configuration initiation operation to send the corresponding database configuration information of the source database and the target database to the terminal device. The terminal device then calls the MySQL database (i.e., the configuration initiation service module mentioned above) to create a data migration job batch based on the database configuration information, and creates a data migration job task table in the data migration job batch, and then creates a data migration job instance table under the data migration job task table. At the same time, the terminal device generates a DJS temporary job based on the data migration job batch. The DJS temporary job extracts scripts (i.e., the data to be migrated) from the source databases such as the historical database, offline platform, and TD database based on the data migration job batch, and loads each script into the Gauss target database.

[0077] Furthermore, in a feasible embodiment, the step of "determining the database connection information between the source database and the target database" in step S10 above may specifically include:

[0078] Step S101: Receive data source configuration information;

[0079] Step S102: Determine the database connection information between the source database and the target database based on the data source configuration information.

[0080] In this embodiment, when a user needs to migrate data between databases with different data structures, the user can input the data source configuration information corresponding to the source database and the target database to be migrated into the terminal device. The terminal device then receives the data source configuration information input by the user and inputs it into the aforementioned data source management module. The data source management module then stores and saves the data source configuration information and determines the database connection relationship between the source database and the target database based on the data source configuration information.

[0081] For example, when a user needs to migrate data between databases with different data structures, the user can input the data source configuration information, such as the IP address, port, database name, username, password, and data source encoding, of the source database (source_db) and the target database (target_db) to the terminal device. The terminal device then receives the data source configuration information of the source database (source_db) and the target database (target_db) and inputs this information to the data source management module. Subsequently, the data source management module generates data source connection information between the source database (source_db) and the target database (target_db) based on the data source configuration information.

[0082] In this way, by collecting the configuration information corresponding to the source database and the target database respectively, and creating database connection information between the source database and the target database based on the source data configuration information, the terminal device can connect data with different data structures, thereby enabling data migration operations to be performed between different databases.

[0083] It should be noted that, in this embodiment and another embodiment, after receiving the data source configuration information such as IP, port, database name, username, password, and data source encoding input by the user, the terminal device can also filter out each password information in the data source configuration and encrypt each password information to ensure the security of the data migration process.

[0084] Furthermore, in this embodiment and another embodiment, after the terminal device creates the data source connection information between the source database source_db and the target database target_db, it can also perform a test connection operation and obtain the test result. Based on the test result, it can determine whether the connection between the source database source_db and the target database target_db has been successfully established. In this way, the user can intuitively determine whether the connection between the source database source_db and the target database target_db has been successfully established.

[0085] Furthermore, in a feasible embodiment, the step of "creating keyword mapping information based on the database connection information" in step S10 above may specifically include:

[0086] Step S103: Determine the first data type corresponding to the source database and the second data type corresponding to the target database based on the database connection information;

[0087] Step S104: Determine the target keyword in the target database corresponding to the data to be migrated from the source database based on the first data type and the second data type;

[0088] Step S105: Create keyword mapping information based on the target keyword;

[0089] It should be noted that the first data type corresponds to the data structure of the source database, and similarly, the second data type corresponds to the data structure of the target database.

[0090] In this embodiment, after obtaining the database connection information, the data source management module determines the first data type corresponding to the source database and the second data type corresponding to the target database based on the configuration information contained in the database connection information. Then, the data source management module determines the target keywords corresponding to each piece of data to be migrated in the source database in the target database based on the first data type and the second data type. Finally, the data source management module creates keyword mapping information corresponding to each piece of data to be migrated based on the target keywords corresponding to each piece of data to be migrated in the target database.

[0091] For example, after obtaining the database connection information, the data source management module determines a first data type corresponding to the source database `source_db` and a second data type corresponding to the target database `target_db` based on the data source configuration information contained in the database connection information, such as IP address, port, database name, username, password, and data source encoding. Then, the data source management module queries multiple data sources `source_key` to be migrated contained in the source database `source_db` based on the first and second data types, thereby determining the target keyword `target_key` corresponding to each of the multiple data sources `source_key` to be migrated in the target database `target_db`. Finally, the data source management module creates keyword mapping information based on the multiple data sources `source_key` to be migrated and their corresponding keyword `target_key`. In this way, the terminal device can determine the target keywords corresponding to each data source `source_key` in the target database, thereby generating the corresponding keyword mapping information.

[0092] Furthermore, in a feasible embodiment, the step of "processing the initial data migration task flow batch to obtain the target data migration task flow batch" in step S20 above may specifically include:

[0093] Step S201: Determine multiple data migration tasks within the initial data migration task flow batch, and generate a target task flow based on the multiple data migration tasks;

[0094] Step S202: Arrange the multiple data migration tasks according to the target task flow to obtain the target data migration task flow batch;

[0095] In this embodiment, after the configuration initiation service module constructs the initial data migration task flow batch, it first creates multiple data migration tasks within the initial data migration task flow batch and orchestrates the multiple data migration tasks in series to generate the target task flow. Then, the configuration initiation service module orchestrates the multiple data migration tasks within the initial data migration task flow batch based on the target task flow to obtain the target data migration task flow batch.

[0096] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of the task creation process involved in an embodiment of the data migration method of this application, as shown below. Figure 4As shown, after constructing the initial data migration task flow batch, the configuration initiation service module first creates multiple data migration tasks, including an automatic target database table generation task, an automatic data migration configuration generation task, an automatic task scheduling workflow configuration generation task, an automatic data migration execution task, and a status update task. The configuration initiation service module then generates a target data migration task flow based on these multiple data migration tasks and a preset migration environment configuration table. Next, the configuration initiation service module concatenates and orchestrates these multiple data migration tasks based on this target data migration task flow and adds them to the initial data migration task flow batch to obtain the target data migration task flow batch. Finally, it executes this target data migration task flow batch by calling a temporary task initiation interface. In this way, the terminal device can determine the specific migration scenario based on the database selection and configure and orchestrate the migration tasks according to the migration scenario, thereby further improving the efficiency of data migration.

[0097] It should be noted that the migration environment configuration table is a task configuration table pre-set by technicians for the migration environment. It can be understood that the migration environment corresponds to the source database and the target database. When the source database connects to different target databases, different migration environments will be generated. The migration environment configuration table is pre-configured by technicians and stored in the storage device of the terminal device so that the terminal device can read the storage device to obtain the migration environment configuration table when needed. Alternatively, the migration environment configuration table can also be uploaded by technicians to a third-party server, so that the third-party server can access the third-party server to download the migration environment configuration table when needed. This application does not limit the configuration content and storage method of the migration environment configuration table.

[0098] Furthermore, in a feasible embodiment, the step of "obtaining target migration data through the target data migration task flow batch and the keyword mapping information" in step S30 above may specifically include:

[0099] Step S301: Modify the data type of the data to be migrated in the source database using the target data migration task flow batch and the keyword mapping information to obtain standard file data, wherein the data type of the standard file data is a third data type;

[0100] Step S302: Based on the keyword mapping information, modify the data type of the standard file data from the third data type to the second data type to obtain the target migration data;

[0101] It should be noted that the third data type is a data type that conforms to a preset standard specification. It can be understood that the standard specification is a data type specification set in advance by technical personnel, and this application does not limit the specific content of the standard specification.

[0102] In this embodiment, after the configuration initiation service module inputs the target data migration task flow batch to the execution monitoring service module, the monitoring service module first executes multiple data migration tasks contained in the target data migration task flow batch to perform extraction operations according to the steps of each of the multiple data migration tasks, thereby extracting the data to be migrated from the source database, and modifying the data to be migrated based on the keyword mapping information to change the data type of the data to be migrated from the first data type to the third data type, thereby obtaining standard file data. Then, the monitoring service module modifies the data type corresponding to the standard file data, i.e. the third data type, to the second data type based on the keyword mapping information, thereby obtaining the target migration file corresponding to the target database, and loading the target migration file into the target table in the target database to complete the data operation.

[0103] For example, after the configuration initiation service module inputs the target data migration task flow batch to the execution monitoring service module, the monitoring service module first executes multiple data migration tasks contained in the target data migration task flow batch, thereby performing extraction operations according to the steps of each of the multiple data migration tasks to extract the database table t_source to be migrated contained in the source database source_db, and based on the aforementioned keyword mapping information, modifies the data type of the multiple data to be migrated contained in the database table t_source from the aforementioned first type to the third data type corresponding to the preset standard specification to obtain a standard data file that conforms to the standard specification. Then, based on the keyword mapping information, the monitoring service module modifies the data type of the standard data file from the third data type to the aforementioned second data type to obtain the target migration data that matches the target database. At the same time, the monitoring service module creates a target database table t_target in the target database target_db and loads the target migration data into the target database table t_target to complete the data migration operation.

[0104] In this way, the terminal device can unify data files of different data types into standard data files and migrate data between different databases in the form of standard data files. Furthermore, by modifying the data type of the standard data file after the migration is completed, the terminal device can ensure that the migrated data conforms to the data structure of its own database.

[0105] Furthermore, in this embodiment and another embodiment, after the monitoring service module modifies the data types of multiple data to be migrated from the first type to the third data type corresponding to the preset standard specification to obtain standard data files that conform to the standard specification, it can first create a target database table t_target in the target database target_db, and load each of the standard data files into the target database table t_target in the target database target_db. Then, the monitoring service module modifies the data type of the standard file data in the target database table t_target based on the keyword mapping information, thereby changing the data type from the third data type to the second data type to obtain the target migration data, and thus completing the data migration operation.

[0106] In addition, please refer to this embodiment and another embodiment. Figure 5 , Figure 5 This is a schematic diagram of the task execution flow involved in an embodiment of the data migration method of this application, as shown below. Figure 5 As shown, when the terminal device executes the target data migration task flow batch, the terminal device can also identify the target data migration task flow batch and determine whether the task in the target data migration task flow batch is a data migration task. Then, if the terminal device determines that the task is a data migration task, it determines the resource allocation scheme corresponding to the target data migration task flow batch, and calls computer resources for the target data migration task flow batch to execute the data migration task based on the resource allocation scheme.

[0107] In addition, please refer to this embodiment and another embodiment. Figure 6 , Figure 6 This is a schematic diagram of the server allocation process involved in an embodiment of the data migration method of this application, as shown below. Figure 6 As shown, when a terminal device performs a data migration task, the terminal device can first initiate a temp_master script process, which then selects a server. Simultaneously, the terminal device obtains a preset terminal device parameter -c and inputs this -c parameter into the temp_master script process. The temp_master script process then initiates a worker process based on this -c parameter to determine the target terminal for performing the data migration task within the server. Finally, the temp_master script process registers a file according to the -f parameter corresponding to the target terminal, so that the target terminal can execute the target data migration task batch.

[0108] Based on the first embodiment of the data migration method of this application described above, a second embodiment of the data migration method of this application is hereby proposed.

[0109] Furthermore, in a feasible embodiment, after the step of "creating an initial data migration task flow batch based on the database connection information and the keyword mapping information" in step S20 above, the data migration method of this application may further include the following steps:

[0110] Step A10: Determine whether the initial data migration task stream batch contains large-volume tasks;

[0111] Step A20: If it is determined that the initial data migration task flow batch contains the large-volume task, then obtain the preset splitting rule, and split the large-volume task into multiple small-volume tasks according to the splitting rule.

[0112] Step A30: Add multiple of the small-volume tasks to the initial data migration task stream batch to obtain the target data migration task stream batch;

[0113] It should be noted that the large-volume task refers to a data migration task involving target database tables with a data volume greater than or equal to a preset data volume threshold. It is understood that the specific value of the data volume threshold can be set by technical personnel, and this application does not impose any restrictions on it. Similarly, the small-volume task refers to a data migration task involving target database tables with a data volume less than the data volume threshold.

[0114] In this embodiment, after constructing the initial data migration task flow batch, the configuration initiation service module can also read multiple data migration tasks containing each data to be migrated within the source data, and determine whether there are large-volume tasks among the multiple data migration tasks. Then, if the configuration initiation service module determines that there are large-volume tasks among the multiple data migration tasks, it obtains the preset task splitting conditions, and splits the large-volume tasks into multiple small-volume tasks based on the task splitting conditions. Finally, the configuration initiation module adds the multiple small-volume tasks into the initial data migration task flow batch to obtain the target data migration task flow batch.

[0115] For example, after constructing the initial data migration task flow batch, the configuration initiation service module can periodically synchronize the migration data tables in the source database. Based on these tables, it can determine whether any large-volume tasks exist within the initial data migration task flow batch. If a large-volume task is identified, the configuration initiation service module obtains a preset task splitting rule and splits it into multiple smaller tasks. Finally, it adds these smaller tasks to the initial data migration task flow batch to obtain the target data migration task flow batch. In this way, the terminal device can promptly identify large data migration tasks during task flow batch creation and split them into smaller tasks, thereby improving the speed of the data migration process.

[0116] Furthermore, in a feasible embodiment, step A10 above may specifically include:

[0117] Step A101: Detect the initial data migration task flow batch to obtain the migration data table corresponding to each of the multiple data migration tasks in the initial data migration task flow batch;

[0118] Step A102: Determine the migration data volume corresponding to each of the migration data tables, and obtain a preset data volume threshold;

[0119] Step A103: Compare each of the migration data volumes with the data volume threshold to obtain multiple comparison results;

[0120] Step A104: When multiple comparison results include a target comparison result whose migration data volume is greater than the data volume threshold, it is determined that the initial data migration task flow batch contains a large-volume task.

[0121] In this embodiment, the configuration initiation service module first detects the initial data migration task flow batch to obtain the migration data tables corresponding to each of the multiple data migration tasks within the initial data migration task flow batch. Then, the configuration initiation service module determines the migration data volume stored in each of the multiple migration data tables. Simultaneously, the configuration initiation service module obtains a data volume threshold preset by the technical personnel. Next, the configuration initiation service module compares each of the multiple migration data volumes with the data volume threshold to obtain multiple comparison results. Finally, the configuration initiation service module reads the multiple comparison results, and when it determines that one of the multiple comparison results contains a target comparison result with a migration data volume greater than the data volume threshold, it identifies the data migration task containing the data table to be migrated corresponding to that target comparison result as a large-volume task. In this way, the terminal device can promptly identify large-volume data migration tasks when creating task flow batches and divide large-volume data migration tasks into multiple smaller-volume data migration tasks, thereby improving the speed of the data migration process.

[0122] Based on the first and / or second embodiments of the data migration method of this application described above, a preferred embodiment of the data migration method of this application is proposed herein.

[0123] Further, please refer to Figure 7 , Figure 7 This is a flowchart illustrating a preferred embodiment of the data migration method of this application, as shown below. Figure 7As shown in this embodiment, since the AS400 system is the primary system used by banks and other financial institutions, when the AS400 system needs to be taken offline, the enterprise needs to migrate the data in the AS400 system to a historical database for backup. At this time, the terminal device first reads the AS400 historical tape backup drive connected to the AS400 system and restores the data to be migrated backed up in the AS400 historical tape backup drive to the AS400 system. Then, the terminal device deploys Kettle to download the data to be migrated from the AS400 system. If the terminal device detects that the data to be migrated downloaded from the AS400 system has failed to download, it re-executes the download task. However, if the terminal device detects that the data to be migrated downloaded from the AS400 system has been successfully downloaded, but the downloaded data is not compatible with the historical tape backup drive, it will not be able to migrate the data. If the required data volume is inconsistent with the preset data volume, the data volume consistency check is determined to have failed, and the download task is re-executed. If the terminal device detects that the data download is successful and the data volume consistency check passes, the data type of the data to be migrated is modified to obtain a standard data file that conforms to the exchange platform standard. At the same time, the terminal device performs a tape backup task on the standard data file to store the standard file data to be backed up in the target tape device. After that, when the terminal device confirms that the tape backup task is successful, it initiates a task cleanup operation, and changes the task execution status to cleanup success upon successful cleanup. Finally, the terminal device determines the target standard file that needs to be stored in the historical database, restores the target standard file in the target tape device to the terminal device, and then loads the target standard file into the historical database, thereby completing the data backup of the AS400 system.

[0124] In addition, to achieve the above objectives, this application also provides a data migration apparatus, please refer to... Figure 8 , Figure 8 This is a schematic diagram of the functional modules involved in an embodiment of the data migration device of this application, as shown below. Figure 8 As shown, the device includes:

[0125] The configuration management module 10 is used to determine the database connection information between the source database and the target database, and to create keyword mapping information based on the database connection information;

[0126] The task creation module 20 is used to create an initial data migration task flow batch based on the database connection information, and process the initial data migration task flow batch to obtain the target data migration task flow batch.

[0127] The task execution module 30 is used to obtain target migration data through the target data migration task flow batch and the keyword mapping information, and load the target migration data into the target database.

[0128] Furthermore, the configuration management module 10 includes:

[0129] Information acquisition unit, used to receive data source configuration information;

[0130] The first creation unit is used to determine the database connection information between the source database and the target database based on the data source configuration information.

[0131] Furthermore, the configuration management module 10 also includes:

[0132] A type identification unit is used to determine, based on the database connection information, the first data type corresponding to the source database and the second data type corresponding to the target database;

[0133] A data matching unit is used to determine the target keyword corresponding to the data to be migrated from the source database in the target database based on the first data type and the second data type.

[0134] The second creation unit is used to create keyword mapping information based on the target keyword.

[0135] Furthermore, the task creation module 20 includes:

[0136] A process generation unit is used to determine multiple data migration tasks within the initial data migration task flow batch, and generate a target task flow based on the multiple data migration tasks.

[0137] The task orchestration unit is used to orchestrate the multiple data migration tasks according to the target task flow to obtain a target data migration task flow batch.

[0138] Furthermore, the task execution module 30 includes:

[0139] The first modification unit is used to modify the data type of the data to be migrated in the source database through the target data migration task flow batch and the keyword mapping information to obtain standard file data, wherein the data type of the standard file data is a third data type;

[0140] The second modification unit is used to modify the data type of the standard file data from the third data type to the second data type based on the keyword mapping information to obtain the target migration data.

[0141] Furthermore, the task creation module 20 also includes:

[0142] The task filtering unit is used to determine whether the initial data migration task flow batch contains large-volume tasks.

[0143] The task splitting unit is used to obtain a preset splitting rule if it is determined that the initial data migration task flow batch contains the large-volume task, and split the large-volume task into multiple small-volume tasks according to the splitting rule.

[0144] The batch modification unit is used to add multiple small-volume tasks to the initial data migration task flow batch to obtain the target data migration task flow batch.

[0145] Furthermore, the task filtering unit includes:

[0146] The data table acquisition subunit is used to detect the initial data migration task flow batch to obtain the migration data table corresponding to each of the multiple data migration tasks in the initial data migration task flow batch.

[0147] The data volume acquisition subunit is used to determine the migration data volume corresponding to each of the migration data tables and to acquire a preset data volume threshold.

[0148] The data volume comparison subunit is used to compare each of the migration data volumes with the data volume threshold to obtain multiple comparison results;

[0149] The result filtering subunit is used to determine that the initial data migration task flow batch contains a large-volume task when multiple comparison results include a target comparison result whose migration data volume is greater than the data volume threshold.

[0150] In addition, this application also provides a terminal device having a computer program that can run on a processor, wherein when the terminal device executes the computer program, it implements the steps of the data migration method as described in any of the above embodiments.

[0151] The specific embodiments of the terminal device in this application are basically the same as the embodiments of the data migration method described above, and will not be repeated here.

[0152] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data migration method as described in any of the above embodiments.

[0153] The specific embodiments of the computer-readable storage medium of this invention are basically the same as the embodiments of the data migration method described above, and will not be repeated here.

[0154] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0155] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile terminal, data storage control terminal, PC, etc., internally configured with a multi-source heterogeneous database end-to-end data migration system) to execute the methods described in the various embodiments of this application.

[0157] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data migration method, characterized in that, The data migration method includes the following steps: Determine the database connection information between the source database and the target database, and create keyword mapping information based on the database connection information. The source database and the target database are databases containing file data of different data types. The database includes an in-warehouse database and an out-of-warehouse database. The in-warehouse database is a database that has recorded configuration information, and the out-of-warehouse database is a database that has not recorded configuration information. An initial data migration task flow batch is created based on the database connection information, and the initial data migration task flow batch is processed to obtain the target data migration task flow batch. The target migration data is obtained by using the target data migration task flow batch and the keyword mapping information, and then the target migration data is loaded into the target database. The step of processing the initial data migration task flow batch to obtain the target data migration task flow batch includes: Identify multiple data migration tasks within the initial data migration task flow batch, and generate a target task flow based on the multiple data migration tasks; The multiple data migration tasks are arranged according to the target task flow to obtain a target data migration task flow batch; The step of obtaining the target migration data through the target data migration task flow batch and the keyword mapping information includes: The data type of the data to be migrated in the source database is modified by the target data migration task flow batch and the keyword mapping information to obtain standard file data, wherein the data type of the standard file data is a third data type; Based on the keyword mapping information, the data type of the standard file data is modified from the third data type to the second data type to obtain the target migration data.

2. The data migration method as described in claim 1, characterized in that, The step of determining the database connection information between the source database and the target database includes: Receive data source configuration information; The database connection information between the source database and the target database is determined based on the data source configuration information.

3. The data migration method as described in claim 1, characterized in that, The step of creating keyword mapping information based on the database connection information includes: Based on the database connection information, determine the first data type corresponding to the source database and the second data type corresponding to the target database; Based on the first data type and the second data type, determine the target keyword corresponding to the data to be migrated from the source database in the target database; Keyword mapping information is created based on the target keywords.

4. The data migration method as described in claim 1, characterized in that, After the step of creating an initial data migration task flow batch based on the database connection information, the method further includes: Determine whether the initial data migration task batch contains large-volume tasks; If it is determined that the initial data migration task flow batch contains the large-volume task, then a preset splitting rule is obtained, and the large-volume task is split into multiple small-volume tasks according to the splitting rule. Multiple small-volume tasks are added to the initial data migration task flow batch to obtain the target data migration task flow batch.

5. The data migration method as described in claim 4, characterized in that, The step of determining whether the initial data migration task stream batch contains large-volume tasks includes: The initial data migration task flow batch is inspected to obtain the migration data table corresponding to each of the multiple data migration tasks within the initial data migration task flow batch. Determine the amount of migration data corresponding to each of the migration data tables, and obtain a preset data volume threshold; The migration data volume of each data point is compared with the data volume threshold to obtain multiple comparison results; When multiple comparison results include a target comparison result where the amount of migrated data is greater than the data volume threshold, it is determined that the initial data migration task flow batch contains a large-volume task.

6. A data migration device, characterized in that, The device includes: The configuration management module is used to determine the database connection information between the source database and the target database, and to create keyword mapping information based on the database connection information. The source database and the target database are databases containing file data of different data types. The databases include an in-warehouse database and an out-of-warehouse database. The in-warehouse database is a database that has recorded configuration information, and the out-of-warehouse database is a database that has not recorded configuration information. The task creation module is used to create an initial data migration task flow batch based on the database connection information, and to process the initial data migration task flow batch to obtain the target data migration task flow batch. The task execution module is used to obtain target migration data through the target data migration task flow batch and the keyword mapping information, and load the target migration data into the target database; The task creation module is also used to determine multiple data migration tasks within the initial data migration task flow batch, and generate a target task flow based on the multiple data migration tasks. The multiple data migration tasks are arranged according to the target task flow to obtain a target data migration task flow batch; The task execution module is further configured to modify the data type of the data to be migrated in the source database through the target data migration task flow batch and the keyword mapping information to obtain standard file data, wherein the data type of the standard file data is a third data type; Based on the keyword mapping information, the data type of the standard file data is modified from the third data type to the second data type to obtain the target migration data.

7. A terminal device, characterized in that, The terminal device includes: a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the computer program, when executed by the processor, implements the steps of the data migration method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data migration method as described in any one of claims 1 to 5.

Images