A data migration system between information systems with complex storage structures

Abstract

The present invention discloses a data migration method for an information system with a complex storage structure, comprising the following steps: initializing after instantiating a database data source, configuring a data transmission project instance; parsing the configuration of the data transmission project instance, and building a job flow according to the configuration; combining the job flow into a plurality of different task instruction batches and executing them; sequentially executing different batches of task instructions according to the type of the task instruction batch, and concurrently executing tasks within the same batch; and updating the execution progress after completing the execution. The data migration method for an information system with a complex storage structure described in the present invention extracts, converts, and loads business data from a system information system at a source end to an information system at a target end under a complex storage structure and complex business demand scenario, and ensures that the migrated data meets the consistency and correctness of the business logic of the target information system.

Description

Technical Field

[0001] The present invention belongs to the field of data migration, and in particular relates to a data migration system between information systems with complex storage structures. Background Art

[0002] As the business of an enterprise develops, the information system that supports the business usually needs to be restructured or rebuilt. In order to ensure the continuity of the business, it is necessary to migrate the business data from the old business system to the new business system. When the amount of data in the information system is large, the data storage structure of the old and new information systems will undergo a major reconstruction, and the existing business data needs to be gradually migrated to the new information system in batches.

[0003] Currently, the more popular data transmission technologies include tools such as Kettle. The working principle of such tools is as follows: (1) Create a data transmission task and use a simple scripting language to configure the data tables and data fields of the relational database on the source side and the data tables and data fields of the relational database on the target side. The data field configuration on the source side can also be replaced by query SQL. This implements the data extraction logic on the source side and the data loading logic on the target side. (2) In the data transmission task, use a simple scripting language to configure the data field mapping relationship between the source side and the target side. This implements the data conversion logic. (3) In the data transmission task, configure the data sources of the relational databases on the source side and the target side, mainly the connection address, user and password. (4) Assemble multiple data transmission tasks into a job flow, set the execution order of data transmission tasks, execute the job flow, and transfer the business data from multiple data tables in multiple databases to the target side. The application scenarios of existing data transmission technologies are mainly data replication between databases and data collection and analysis on data platforms. When faced with complex storage structures and complex business requirements, they have the disadvantages of insufficient functional support and high usage costs.

[0004] In view of this, there is an urgent need to propose a data migration system between information systems with complex storage structures. Summary of the invention

[0005] To this end, the present invention provides a data migration method and a data migration system between information systems with complex storage structures, which extract, convert, and load business data from the system information system on the source side to the information system on the target side under complex storage structures and complex business demand scenarios, and ensure that the migrated data meets the consistency and correctness of the business logic of the target information system.

[0006] The above object of the present invention is achieved through the following technical solutions:

[0007] A data migration method between information systems with complex storage structures comprises the following steps:

[0008] Initialize the database data source after instantiating it, and configure a data transfer project instance;

[0009] Parse the configuration of the data transfer project instance and build the job flow according to the configuration;

[0010] Combine the work flow into multiple different task instruction batches for execution; according to the type of task instruction batches, execute the task instructions of different batches in sequence, and execute the tasks in the same batch concurrently;

[0011] Update the execution progress after completing the execution.

[0012] Furthermore, a data transmission project corresponds to multiple instances, and each instance is uploaded to the data transmission project work pool for operation; when the work pool is full, new instances are rejected.

[0013] Furthermore, the configuration data, project variable properties, data partition configuration and task configuration registry of a data transmission project are combined and configured into an instance.

[0014] Furthermore, the data transmission workflow is constructed, including:

[0015] Parse and load data partition information;

[0016] Loading task configuration; the task configuration includes global data task configuration and partition data task configuration;

[0017] Configure the execution order based on the business logic within the task.

[0018] Furthermore, when the job flow is combined into a plurality of different task instruction batches, the task instruction batches are combined in the order of global data tasks, partitioned data tasks and task dependencies.

[0019] Furthermore, the instance of the data transmission project stores the execution logic of data extraction, conversion and loading, and the source and target ends of the data are not specified in each execution logic; the task instructions in the job flow specify the source and target ends of the data and perform read and write operations.

[0020] Furthermore, it also includes pre-processing and post-processing;

[0021] The pre-processing is used to check the idempotence of the job after initialization;

[0022] The post-processing is used to check the data consistency before and after the data migration before the execution is completed.

[0023] The present invention also provides a data migration system between information systems with complex storage structures, comprising:

[0024] A data source management module is used to configure data source information including single data source and multiple data sources, and perform data partition management;

[0025] The transfer task management module manages the transfer tasks including data migration, data export and data import;

[0026] The data transmission project management module manages the preset data transmission projects.

[0027] The present invention also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, a migration method between information systems with complex storage structures is implemented.

[0028] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a migration method between information systems with a complex storage structure is implemented.

[0029] The above technical solution of the present invention has the following advantages compared with the prior art:

[0030] In terms of execution, the present invention reuses job task instances at each active node, starts allocating resources when the job task instance is running, and recycles resources in time when the job task instance ends running. The database connection of the same database instance is reused through a database switching command. Through batch reading and writing of the database, streaming reading and writing of data files achieves efficient data transmission with low memory usage. At the same time, the workload of job development and the resource consumption during operation of the present invention are greatly reduced compared with the existing technical solutions, and the gap will expand rapidly with the increase of the number of library tables and the number of business units. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of the structure of a system provided by an embodiment of the present invention;

[0032] Figure 2 is a flow chart of a method provided by an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and systems can be simplified for display.

[0035] The following description and accompanying drawings fully illustrate specific embodiments of the present invention so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The examples represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operations may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. The scope of the embodiments of the present invention includes the entire scope of the claims, and all available equivalents of the claims. In this article, each embodiment may be represented individually or generally by the term "invention", which is only for convenience, and if more than one invention is disclosed in fact, it is not intended to automatically limit the scope of the application to any single invention or inventive concept. In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. Moreover, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method or electronic device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or electronic device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method or electronic device including the elements. The various embodiments herein are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the methods, products, etc. disclosed in the embodiments, since they correspond to the method part disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0036] Embodiment 1

[0037] Please refer to Figure 1 As shown, a data migration method between information systems with complex storage structures includes the following steps:

[0038] Initialize the database data source after instantiating it, and configure a data transfer project instance;

[0039] Parse the configuration of the data transfer project instance and build the job flow according to the configuration;

[0040] Combine the work flow into multiple different task instruction batches for execution; according to the type of task instruction batches, execute the task instructions of different batches in sequence, and execute the tasks in the same batch concurrently;

[0041] Update the execution progress after completing the execution.

[0042] Furthermore, a data transmission project corresponds to multiple instances, and each instance is uploaded to the data transmission project work pool for operation; when the work pool is full, new instances are rejected.

[0043] Furthermore, the configuration data, project variable properties, data partition configuration and task configuration registry of a data transmission project are combined and configured into an instance.

[0044] Furthermore, the data transmission workflow is constructed, including:

[0045] Parse and load data partition information;

[0046] Loading task configuration; the task configuration includes global data task configuration and partition data task configuration;

[0047] Configure the execution order based on the business logic within the task.

[0048] Furthermore, when the job flow is combined into a plurality of different task instruction batches, the task instruction batches are combined in the order of global data tasks, partitioned data tasks and task dependencies.

[0049] Furthermore, the instance of the data transmission project stores the execution logic of data extraction, conversion and loading, and the source and target ends of the data are not specified in each execution logic; the task instructions in the job flow specify the source and target ends of the data and perform read and write operations.

[0050] Furthermore, it also includes pre-processing and post-processing;

[0051] The pre-processing is used to check the idempotence of the job after initialization;

[0052] The post-processing is used to check the data consistency before and after the data migration before the execution is completed.

[0053] Please refer to Figure 2 As shown, this embodiment also provides a data migration system between information systems with complex storage structures, including:

[0054] A data source management module is used to configure data source information including single data source and multiple data sources, and perform data partition management;

[0055] A data source management module is used to configure data source information including single data source and multiple data sources, and perform data partition management;

[0056] The transfer task management module manages the transfer tasks including data migration, data export and data import;

[0057] The data transmission project management module manages the preset data transmission projects.

[0058] Embodiment 2

[0059] The system of this embodiment includes a data source management module, a transmission task management module and a data transmission project management module.

[0060] The data source management module includes database data source configuration and data partition configuration.

[0061] Database data source configuration supports single data source configuration and multiple data source configuration. Multiple data source configuration consists of a group of data source configurations. A data source configuration includes data source ID, database connection, user name, and password. The implementation of database data source uses database connection pool. Database data source is independent of data transmission project and can be referenced by multiple data transmission projects and transmission tasks, maximizing the reuse of data source and database connection.

[0062] Data partition configuration is used to migrate data in batches by data partition. The items of a data partition configuration include partition ID, partition name, read database, write database, read path, and write path. When migrating data by business unit, a business unit is used as a data partition, the unique identifier of the business unit is used as the partition ID, and the business unit name is used as the partition name. When the business data of the source end or the target end is horizontally split and stored in multiple sub-databases or multiple data file directories, the corresponding read database name, write database name, read file path, and write file path are configured respectively.

[0063] The transfer task management module manages the execution of each transfer task, including general tasks, data migration tasks, data export tasks, and data import tasks. General tasks are used for auxiliary operations such as notification of processing results, downloading of data files, and uploading of data files. They are not used for the migration of business data between the source and target ends. Data migration tasks are used to migrate business data from the database on the source end to the database on the target end. Data export tasks are used to migrate business data from the database on the source end to the data files on the target end. Data import tasks are used to migrate business data from the data files on the source end to the database on the target end.

[0064] The data transmission project to be transmitted in this embodiment is a specific implementation of a data migration workflow, which is used to schedule and execute data transmission tasks, record detailed execution progress, and execute the final processing logic after the workflow is completed. When constructing a data transmission project, the main components include a project initializer, a task configuration registry, a task factory, a task executor, a data transmission project progress management, and a project final processor.

[0065] In the system of this embodiment, multiple data transmission projects may exist simultaneously, and each data transmission project may have multiple running instances. The running instances are submitted to the data transmission project manager in the system of this embodiment for running.

[0066] This embodiment takes a data transmission project as an example and specifically implements the method described in this embodiment.

[0067] The system of this embodiment first instantiates a database data source. It should be noted that the data source is shared by all instances of data transmission projects. The data source instance uses a database connection pool to manage database connections. Only a minimum number of connections are created during initialization, and then database connections are created and closed according to operational needs.

[0068] In the project scheduling stage, use the data transfer project assembler to assemble the project configuration properties, project variable properties, data partition configuration, and task configuration registry into an instance of a data transfer project. Then use the data transfer project manager to submit the data transfer project. The submission will be successful only when the work pool has sufficient resources, otherwise the data project submission will be rejected. Additional operating parameters can be set when submitting a data transfer project.

[0069] At this time, the data transfer project instance simply integrates the assembly configuration and does not occupy the business database connection, business service network connection, and server hard disk.

[0070] Then the data transmission project is initialized. In the initialization phase, the preparation work of parsing the project configuration properties, project variable properties, and running parameters is first performed. Then the data transmission job flow is built. The data partition information is parsed and loaded from the data partition configuration. The task configuration is loaded from the task configuration registry. The task configuration is divided into two groups: global data task configuration and partition data task configuration. The priority of the global data task group is higher than the partition data task group. The tasks in each group are executed in order according to the business logic configuration, and the task factory is used to create task instances according to the task configuration. Finally, the job flow task instructions are constructed according to the task group list and the data partition information list. Each job task instruction corresponds to an activity node in the job flow. The job flow task instructions and the overall status of the project constitute the execution progress of the data transmission project.

[0071] The data transmission task instance carries the specific execution logic of data extraction, conversion, and loading. It is not bound to the specific source and target end read database, write database, read data file, and write data file. It can be reused by all data partition jobs, and there is no need to create a task instance for each node in the job flow. The task job instruction carries the source and target end read database, write database, read data file, and write data file identification and the execution status of the job. It is a simple data structure that takes up very little memory.

[0072] First, the task instructions of the operation process are combined into task instruction batches in the order of global data tasks, partitioned data tasks, and task dependencies. Tasks in the same batch have no business dependencies and are executed concurrently after batch submission using the task executor. After the task batch is successfully executed, the next batch will be executed. If all task batches are executed successfully or the task execution fails, the project will be ended.

[0073] Finally, the data transfer project instance will end the operation process, update the execution progress, and release the resources occupied by the project. At this time, the data transfer operation process has ended. Finally, the callback executes the code of the final processor, which is usually used to implement additional processing logic such as result notification and hard disk file cleanup.

[0074] Next, this embodiment is described by taking the business data migration requirements of the SaaS system as an example.

[0075] The business data storage structure of the preset SaaS system uses tenants as business units. Business data is stored in 5 database instances. The core business data of a business unit is assigned to a sub-database. A business unit has 100 business tables. The number of database tables on the target end is the same. If the business data of 300 business units is migrated as a whole at one time, the number of active nodes in the data migration operation process is 30,000.

[0076] The solution requires writing 30,000 job task configurations, each of which is configured with a source data source and a target data source. When the job process is running, a maximum of 30,000 task instances and 60,000 database connections need to be loaded. The development workload is huge, and the resource consumption during job runtime is huge and it is very easy to exceed the maximum resource limit, which makes it difficult to effectively implement batch migration of large amounts of business data by business unit.

[0077] By adopting the embodiment of the present invention, only 10 data source configurations, 100 job task configurations and 300 lines of data partition configuration need to be written. Through the reuse of job task instances and database connections, a maximum of 100 task instances, 30,000 lines of job instructions and 200 database connections need to be loaded when the job flow is running.

[0078] In the execution of the job flow, this embodiment reuses the job task instance when the active node of the process is executed, starts to allocate resources when the job task instance is running, and recycles resources in time when the job task instance ends. The database connection of the same database instance is reused through the database switching command. Through batch reading and writing of the database, the streaming reading and writing of data files achieves efficient data transmission with low memory usage.

[0079] Embodiment 3

[0080] Please refer to Figure 3 As shown, the embodiment of the present disclosure provides an electronic device for data migration between information systems with complex storage structures, including a processor (processor) 30 and a memory (memory) 31. Optionally, the electronic device may also include a communication interface (CommunicationInterface) 32 and a bus 33. Among them, the processor 30, the communication interface 32, and the memory 31 can communicate with each other through the bus 33. The communication interface 32 can be used for information transmission. The processor 30 can call the logic instructions in the memory 31 to execute the data migration method between information systems with complex storage structures of the above embodiment.

[0081] An embodiment of the present disclosure provides a storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the data migration method between information systems with the complex storage structure.

[0082] The above-mentioned storage medium may be a transient computer-readable storage medium or a non-transient computer-readable storage medium. Non-transient storage media include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, and may also be a transient storage medium.

[0083] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or electronic device including the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0084] This embodiment is written in a mainstream development language and introduces only minimal basic dependencies. It can be developed as an independent application or easily integrated into other applications.

[0085] This embodiment provides a comprehensive default configuration to support the rapid writing of data transmission task configuration and data transmission project configuration.

[0086] The workload of job development and the resource consumption at runtime of this embodiment are greatly reduced compared with the existing embodiment scheme, and the gap will expand rapidly as the number of libraries and tables and the number of business units increase.

[0087] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, systems and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0088] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A data migration method between information systems with complex storage structures, characterized in that: The steps include: Initialize the database data source after instantiating it, and configure a data transfer project instance; Parse the configuration of the data transfer project instance and build the job flow according to the configuration; Combine the work flow into multiple different task instruction batches for execution; according to the type of task instruction batches, execute the task instructions of different batches in sequence, and execute the tasks in the same batch concurrently; Update the execution progress after completing the execution; One data transfer project corresponds to multiple instances. Each instance is uploaded to the data transfer project work pool for operation. When the work pool is full, new instances are rejected. Combining the configuration data, project variable properties, data partition configuration and task configuration registry of a data transfer project into an instance; Build a data transfer workflow, including: Parse and load data partition information; Loading task configuration; the task configuration includes global data task configuration and partition data task configuration; Configure the execution order according to the business logic within the task; When combining the job flow into multiple different task instruction batches, the task instruction batches are combined in the order of global data tasks, partition data tasks and task dependencies; The instance storage of the data transfer project performs the execution logic of data extraction, conversion, and loading, and the source and target of the data are not specified in each execution logic; the task instructions in the job flow specify the source and target of the data and perform read and write operations.

2. According to the data migration method between information systems with complex storage structures as described in claim 1, it is characterized in that: It also includes pre-processing and post-processing; The pre-processing is used to check the idempotence of the job after initialization; The post-processing is used to check the data consistency before and after the data migration before the execution is completed.

3. A data migration system between information systems with complex storage structures, characterized in that: include: A data source management module is used to configure data source information including single data source and multiple data sources, and perform data partition management; The transfer task management module manages the transfer tasks including data migration, data export and data import; Data transmission project management module, which manages the preset data transmission projects; One data transfer project corresponds to multiple instances. Each instance is uploaded to the data transfer project work pool for operation. When the work pool is full, new instances are rejected. Combining the configuration data, project variable properties, data partition configuration and task configuration registry of a data transfer project into an instance; Build a data transfer workflow, including: Parse and load data partition information; Loading task configuration; the task configuration includes global data task configuration and partition data task configuration; Configure the execution order according to the business logic within the task; When combining the job flow into multiple different task instruction batches, the task instruction batches are combined in the order of global data tasks, partition data tasks and task dependencies; The instance storage of the data transfer project performs the execution logic of data extraction, conversion, and loading, and the source and target of the data are not specified in each execution logic; the task instructions in the job flow specify the source and target of the data and perform read and write operations.

4. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 2 is implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images