Method and device for automatically migrating data

Abstract

The invention discloses a method and a device for automatically migrating data, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: determining a current migration link according to a plurality of migration links of a current migration item and an execution sequence among the migration links, and creating a migration link task corresponding to the current migration link; creating a current migration module task in the migration link tasks by using a data module contained in each pre-configured migration link; obtaining to-be-migrated detail data of the current migration module task from an emigration party, and storing the obtained to-be-migrated detail data in a database; writing the to-be-migrated detail data in the database into an immigration party; after writing is completed, comparing the detail data to be migrated between the database and the immigration party; and monitoring the migration state of the to-be-migrated detail data, and obtaining the execution condition of the current migration module task according to the migration state. According to the embodiment, a universal migration solution which is automatically executed, reusable and high in performance can be provided.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for automatically migrating data. Background Art

[0002] As the business grows, computer systems are constantly updated and iterated. When the original system cannot support the new business, a new system is generally rebuilt, and then the business data of the original system is migrated to the new system. In the existing technology, the method of data migration is to export data from the original system and then initialize it in the new system, or the original system calls the interface of the new system and initializes it through manual triggering. These two methods can be used when the data volume is small. Once the data volume reaches tens of millions, the availability will decrease. Moreover, if multiple systems need to be migrated to the new system, each system needs to invest product, R&D, and testing resources for development, which cannot be accumulated and reused, nor can it form professional migration components and reduce the marginal human cost of multiple migrations. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for automatically migrating data, which can provide an automatic, reusable, high-performance universal migration solution for actual data migration scenarios.

[0004] To achieve the above objective, according to one aspect of the present invention, a method for automatically migrating data is provided.

[0005] The method for automatically migrating data in an embodiment of the present invention includes: determining the current migration link of the current migration project based on multiple migration links of the pre-configured current migration project and the execution order between the migration links, and creating a migration link task corresponding to the current migration link; using the data modules included in the pre-configured migration links to create the current migration module task in the migration link task; obtaining the detailed data to be migrated of the current migration module task from the migrating party, and storing the obtained detailed data to be migrated in a preset database; writing the detailed data to be migrated in the database to the migrating party; after the writing is completed, performing a comparison of the detailed data to be migrated between the database and the migrating party; monitoring the migration status of the detailed data to be migrated, and obtaining the execution status of the current migration module task based on the migration status.

[0006] Optionally, obtaining the detailed data to be migrated of the current migration module task from the migrating party includes: determining input parameters of the migrating party's migration interface based on pre-acquired description information of the detailed data to be migrated, and calling the migration interface using the input parameters to obtain the detailed data to be migrated.

[0007] Optionally, the method further includes: obtaining output parameters of the migration-out interface before, after or while using the input parameters to call the migration-out interface to obtain the detailed data to be migrated; constructing input parameters of the migration-in interface of the migrating party based on the output parameters, and storing the input parameters of the migration-in interface in the database.

[0008] Optionally, writing the detailed data to be migrated in the database to the migrating party includes: in response to the preset migration execution conditions being met, reading the input parameters of the migration interface from the database, and calling the migration interface using the input parameters of the migration interface to write the detailed data to be migrated in the database to the migrating party.

[0009] Optionally, the comparison of the detailed data to be migrated between the database and the migrating party includes: calling the query interface of the migrating party using the input parameters of the migrating interface to obtain the detailed data after migration; calling the query interface of the database according to the input parameters of the migrating interface to obtain the detailed data before migration; and comparing whether the detailed data after migration is consistent with the detailed data before migration.

[0010] Optionally, the method further includes: after creating the current migration module task, generating a module task creation message and sending it to a preset message queue; the module task creation message is used to trigger the acquisition of the detailed data to be migrated in the migrating party; after storing the detailed data to be migrated in the database, generating a preparation completion message and sending it to the message queue; the preparation completion message is used to trigger the writing of the detailed data to be migrated; after writing the detailed data to be migrated to the migrating party, generating a migration completion message and sending it to the message queue; the migration completion message is used to trigger the comparison of the detailed data to be migrated; after the comparison is completed, generating a comparison completion message and sending it to the message queue.

[0011] Optionally, monitoring the migration status of the detailed data to be migrated includes: obtaining the module task creation message, the preparation completion message, the migration completion message and the comparison completion message from the message queue, and determining the migration status of the detailed data to be migrated based on the module task creation message, the preparation completion message, the migration completion message and the comparison completion message.

[0012] Optionally, the method further includes: pre-establishing multiple secondary queues for storing the migration status of the detailed data to be migrated and a primary queue for storing the identifier of the secondary queue; in response to updating the migration status of any detailed data to be migrated, obtaining the corresponding secondary queue identifier from the primary queue according to the identifier of the detailed data to be migrated, and locating the detailed data to be migrated in the secondary queue indicated by the secondary queue identifier.

[0013] Optionally, obtaining the execution status of the current migration module task based on the migration status includes: when it is monitored that the migration status of any detailed data to be migrated of the current migration module task is execution failure, determining the execution status of the current migration module task as failure; when it is monitored that the migration status of each detailed data to be migrated of the current migration module task is execution success, determining the execution status of the current migration module task as success.

[0014] Optionally, the method further includes: when the execution status of the current migration module task is a failure, issuing a migration exception reminder; when the execution status of the current migration module task is a success, creating the next migration module task of the migration link task and automatically executing it; when the execution status of each migration module task in the migration link task is a success, determining the next migration link of the current migration project according to the execution order between the migration links, creating the migration link task corresponding to the next migration link and automatically executing it.

[0015] To achieve the above objective, according to another aspect of the present invention, a device for automatically migrating data is provided.

[0016] The device for automatically migrating data in an embodiment of the present invention includes: a general control unit, which is used to determine the current migration link of the current migration project based on multiple migration links of the pre-configured current migration project and the execution order between the migration links, and create a migration link task corresponding to the current migration link; a task creation unit, which is used to create the current migration module task in the migration link task by using the data modules included in the pre-configured migration links; a task detail preparation unit, which is used to obtain the detailed data to be migrated of the current migration module task from the migrating party, and store the obtained detailed data to be migrated in a preset database; a task detail execution migration unit, which is used to write the detailed data to be migrated in the database to the migrating party; a task detail comparison unit, which is used to perform a comparison of the detailed data to be migrated between the database and the migrating party after the writing is completed; a task monitoring and reporting unit, which is used to monitor the migration status of the detailed data to be migrated, obtain the execution status of the current migration module task according to the migration status, and report it to the general control unit.

[0017] To achieve the above objective, according to another aspect of the present invention, an electronic device is provided.

[0018] An electronic device of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method of automatically migrating data provided by the present invention.

[0019] To achieve the above objective, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0020] A computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the method for automatically migrating data provided by the present invention.

[0021] According to the technical solution of the present invention, the embodiments of the above invention have the following advantages or beneficial effects:

[0022] For actual data migration scenarios, an automatic, reusable, high-performance, and highly scalable general data migration framework is provided. The data migration framework includes a master control unit, a task creation unit, a task detail preparation unit, a task detail execution and migration unit, a task detail comparison unit, and a task monitoring and reporting unit. The preparation, execution, and comparison processes of the detailed data to be migrated are automatically executed in a task-driven mode with complete decoupling between the units. For specific migration projects, only the configuration of each migration link and its execution order and the determination of relevant parameters of the detailed data can be triggered with one click in the master control unit, thereby realizing the automatic creation and efficient execution of migration tasks of various granularities, reducing manual intervention and lowering the labor costs of products, R&D, testing, etc. At the same time, the task execution status of the entire migration process is automatically reported by the task monitoring and reporting unit, which helps to detect anomalies in a timely manner and then resolve them through manual intervention.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0025] Figure 1 This is a schematic diagram of the main steps of the method for automatically migrating data in an embodiment of the present invention;

[0026] Figure 2 is a schematic diagram of the overall architecture of the first embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of the execution flow of the master control unit of the first embodiment of the present invention;

[0028] Figure 4 1 is a schematic diagram of the execution flow of the task creation unit of the first embodiment of the present invention;

[0029] Figure 5 is a schematic diagram of the execution flow of the task detail preparation unit of the first embodiment of the present invention;

[0030] Figure 6 1 is a schematic diagram of the execution flow of the task detail execution migration unit according to the first embodiment of the present invention;

[0031] Figure 7 1 is a schematic diagram of the execution flow of the task detail comparison unit of the first embodiment of the present invention;

[0032] Figure 8 1 is a schematic diagram of the execution flow of the task monitoring and reporting unit of the first embodiment of the present invention;

[0033] Figure 9 Schematic diagram of components of an apparatus for automatic data migration according to an embodiment of the present invention;

[0034] Figure 10 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0035] Figure 11 2 is a schematic diagram of the structure of an electronic device used to implement the method for automatically migrating data in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0038] Figure 1 1 is a schematic diagram of the main steps of a method for automatically migrating data according to an embodiment of the present invention.

[0039] like Figure 1As shown, the automated data migration method of an embodiment of the present invention is executed by a server for performing data migration, and can migrate data from the migrating party to the migrating party. The migrating party refers to the computer system that serves as the data source in the data migration, and can be a single server or a computer cluster composed of multiple servers. The migrating party refers to the computer system that serves as the data migration destination, and can also be a single server or a computer cluster composed of multiple servers. The automated data migration method specifically includes the following steps:

[0040] Step S101: determining a current migration link of the current migration project according to a plurality of pre-configured migration links of the current migration project and the execution order between the migration links, and creating a migration link task corresponding to the current migration link.

[0041] The above migration links refer to the multiple stages that a migration project needs to go through. For example, a migration project includes migration links such as the pre-stage link, the formal stage, and the supplementary stage. The pre-stage link is used to migrate some basic data, the formal stage is used to migrate the main data of the project, and the supplementary stage is used to supplement after the main data migration is completed. In this step, the server can determine the current migration link to be executed and create a corresponding migration link task for it based on the multiple migration links pre-configured by the staff and the execution order between the migration links. The above migration links can also be called migration types, and the migration link tasks can also be called master control tasks. The execution order between the migration links can represent the order of the migration links during the execution of the migration project.

[0042] Specifically, when performing the first data migration for the current migration project, the server may determine the first migration link among multiple migration links as the current migration link, where the first link is indicated by the execution order of the migration links. If the first migration link executes successfully, the server may determine the second migration link based on the execution order, designate the second migration link as the current migration link, and execute the data migration. This process continues in this manner until each migration link is completed. In actual applications, the server can obtain information about the execution status of each migration link and its modules through the task-driven and asynchronous messaging mechanisms described later, thereby automatically triggering and executing subsequent tasks.

[0043] Step S102: creating a current migration module task in the migration stage task using the pre-configured data modules included in each migration stage.

[0044] The above data modules refer to different areas of business data, such as user modules, merchant modules, and order modules. In practice, staff can pre-configure the data modules to be migrated for each migration phase, and the server will create the corresponding migration module tasks based on these configurations. By configuring migration methods and creating tasks from migration phases to migration modules, the overall migration project can be broken down into smaller subtasks, thereby improving migration efficiency and ensuring a high migration success rate.

[0045] Step S103: Obtain the detailed data to be migrated of the current migration module task from the migrating party, store the obtained detailed data to be migrated in a preset database; write the detailed data to be migrated in the database to the migrating party; after the writing is completed, perform a comparison of the detailed data to be migrated between the database and the migrating party.

[0046] This step is used to perform data preparation, migration execution, and pre- and post-migration data comparison during the migration process. In an embodiment of the present invention, the specific method of data preparation can be that the server determines the input parameters of the outgoing interface of the migrating party based on the description information of the detailed data to be migrated obtained in advance, and uses the above input parameters to call the outgoing interface to obtain the detailed data to be migrated. In a specific application, before, after, or at the same time as obtaining the detailed data to be migrated, the server first obtains the output parameters of the outgoing interface. Since the output parameters of the outgoing interface have a certain correspondence with the input parameters of the incoming interface of the migrating party, the input parameters of the incoming interface of the migrating party can be constructed based on the output parameters of the outgoing interface, and the input parameters of the incoming interface can be stored in the database.

[0047] Preferably, the migration execution method may be as follows: when a preset migration execution condition is met (e.g., the data upon which the data to be migrated depends has been migrated), the server reads the input parameters of the migration-in party's migration-in interface from the database, and calls the migration-in interface using the input parameters of the migration-in interface to write the detailed data to be migrated from the database to the migration-in party. The data comparison method may be as follows: the server calls the migration-in party's query interface using the input parameters of the migration-in interface to obtain the detailed data after migration; calls the database's query interface based on the input parameters of the migration-in interface to obtain the detailed data before migration; and compares the detailed data after migration with the detailed data before migration to ensure consistency.

[0048] The above three migration processes of data preparation, migration execution and comparison of data before and after migration can be automatically triggered and executed in a task-driven manner. In an embodiment of the present invention, after creating the current migration module task, the server generates a module task creation message and sends it to a preset message queue. The module task creation message is used to trigger the acquisition of the detailed data to be migrated in the outgoing party; after storing the detailed data to be migrated in the database, the server generates a preparation completion message and sends it to the message queue. The preparation completion message is used to trigger the writing of the detailed data to be migrated; after writing the detailed data to be migrated to the incoming party, the server generates a migration completion message and sends it to the message queue. The migration completion message is used to trigger the comparison of the detailed data to be migrated. After the comparison is completed, the server generates a comparison completion message and sends it to the message queue. The comparison completion message can trigger the preparation process of the next detailed data.

[0049] Step S104: monitoring the migration status of the detailed data to be migrated, and obtaining the execution status of the current migration module task according to the migration status.

[0050] In this step, the server can obtain the above module task creation message, preparation completion message, migration completion message, and comparison completion message from the message queue, and determine the migration status of the detailed data to be migrated based on the module task creation message, preparation completion message, migration completion message, and comparison completion message. The above migration status can include: task created, preparing, preparation completed (i.e., preparation successful), migration in progress, migration execution completed (i.e., migration execution successful), comparing, comparison completed (i.e., comparison successful). If a failure occurs in the middle, the process is terminated, and the migration status of the detailed data can be: task creation failure, preparation failure, migration execution failure, comparison failure.

[0051] Preferably, the server can pre-establish multiple secondary queues for storing the migration status of the detailed data to be migrated, as well as primary queues for storing the identifiers of the secondary queues. When the migration status of any detailed data to be migrated needs to be updated, the server can retrieve the corresponding secondary queue identifier from the primary queue based on the identifier of the detailed data to be migrated, and locate the detailed data to be migrated in the secondary queue indicated by the secondary queue identifier. This can speed up data queries and help improve the data processing performance of the overall migration architecture.

[0052] In specific applications, the server can determine the execution status of the corresponding migration module task based on the migration status of the monitored detailed data. For example, when the migration status of any detailed data to be migrated of the current migration module task is monitored to be execution failure, the server determines the execution status of the current migration module task as failure; when the migration status of each detailed data to be migrated of the current migration module task is monitored to be execution success, the server determines the execution status of the current migration module task as success. In actual scenarios, when the execution status of the current migration module task is failure, the server issues a migration exception reminder and requests manual intervention to resolve it; when the execution status of the current migration module task is success, the server can create the next migration module task of the migration link task and execute it automatically; when the execution status of each migration module task in the migration link task is success, the server can determine the next migration link of the current migration project according to the execution order between the migration links, create the migration link task corresponding to the next migration link and execute it automatically. In this way, the automatic triggering and orderly execution of multiple tasks under task drive are realized, reducing manual intervention.

[0053] In the technical solution of the embodiment of the present invention, an automatic, reusable, high-performance, and highly scalable general data migration framework is provided for actual data migration scenarios. The data migration framework includes a master control unit, a task creation unit, a task detail preparation unit, a task detail execution migration unit, a task detail comparison unit, and a task monitoring and reporting unit. The preparation, execution, and comparison processes of the detailed data to be migrated are automatically executed in a task-driven mode with complete decoupling between the units. For specific migration projects, it is only necessary to configure the migration links and their execution order and determine the relevant parameters of the detailed data, which can be triggered with one click in the master control unit, thereby realizing the automatic creation and efficient execution of migration tasks of various granularities, reducing manual intervention and lowering the labor costs of products, R&D, testing, etc. At the same time, the task execution status of the entire migration process is automatically reported by the task monitoring and reporting unit, which helps to detect anomalies in a timely manner and then resolve them through manual intervention.

[0054] The first embodiment of the present invention is described below. Figures 2 to 8 . Figure 2 is a schematic diagram of the overall architecture of the first embodiment of the present invention, Figure 3 is a schematic diagram of the execution flow of the master control unit of the first embodiment of the present invention, Figure 4 is a schematic diagram of the execution flow of the task creation unit of the first embodiment of the present invention, Figure 5 is a schematic diagram of the execution flow of the task detail preparation unit of the first embodiment of the present invention, Figure 6 1 is a schematic diagram of the execution flow of the task detail execution migration unit according to the first embodiment of the present invention. Figure 7 1 is a schematic diagram of the execution flow of the task detail comparison unit of the first embodiment of the present invention. Figure 8It is a schematic diagram of the execution flow of the task monitoring and reporting unit of the first embodiment of the present invention.

[0055] As business grows, systems are constantly updated and iterated. When the original system cannot support the new business, a new system is typically built and the business migrated to it. Existing technologies typically export data from the original system and initialize it in the new system, or the original system calls the new system's interface and initializes it manually. These two methods are suitable for small data volumes, but become unwieldy once the data volume reaches tens of millions. Furthermore, if multiple systems need to be migrated to the new system, each system must invest in product development, R&D, and testing. Without the accumulation and reuse of resources, it is impossible to form a professional migration team and reduce the marginal labor costs of multiple migrations.

[0056] The shortcomings of existing technologies are as follows: 1. Migration plans vary from system to system, requiring redesign and development from scratch each time, resulting in high labor costs. 2. Migration across systems lacks consistent experience and reuse, leading to repeated pitfalls and low efficiency. 3. The degree of automation within each system's module migration is low, requiring manual intervention and preventing efficient automated execution.

[0057] The technical problems to be solved in this embodiment are as follows: 1. A unified migration solution, with each system implemented through a task-driven and highly scalable architecture, reducing personnel costs. 2. Common logic is consolidated to improve reusability, avoid pitfalls, and increase efficiency. 3. Each module executes the preparation, migration, and comparison steps in a sequential order and automatically creates the next task, reducing manual intervention and achieving efficient automated execution.

[0058] As can be seen from the system schematic diagram, this method includes 6 modules: general control unit, task creation unit, task detail preparation unit, task detail execution migration unit, task detail comparison unit, and task monitoring and reporting unit. The task drive is divided into two sections: general control task (i.e., migration link task) and module task (i.e., migration module task). The general control task is responsible for overall rhythm control, and the module task is responsible for migrating specific modules. They are explained as follows. Figure 2 In the figure, the solid line represents the flow of the master control task, and the dotted line represents the flow of the module task.

[0059] 1: Master control unit

[0060] The master control unit is the control unit in the entire automated migration process. It mainly includes two parts: system configuration and task creation and execution. The master control unit exists independently and is completely decoupled from the specific migration modules.

[0061] 1. System Configuration

[0062] To ensure the scalability and flexibility of the system, the content related to creating and executing tasks is extracted into configurations. When the business changes, only the configuration needs to be changed. The configuration is isolated according to different business lines, and different business lines have different configurations.

[0063] The configuration includes: the execution order of each migration type (i.e., migration link) and the data module of each migration type. These two configurations can be used to create tasks and automatically execute tasks.

[0064] 2. Task creation and execution

[0065] First, manually create the startup task of the first step, and then the general control personnel issue the task (equivalent to turning on the switch, subsequent tasks are automatically created, issued, and executed), and the task information will be sent to MQ (message queue).

[0066] After receiving the task information, each migration unit executes its own migration logic and reports the migration result (success or failure). After receiving the migration result, the master control updates the status of the corresponding task.

[0067] If the task status is not successful, continue to wait for each migration unit to report. If the task status is successful, it means that the data corresponding to the task has been successfully migrated.

[0068] Determine whether the task is configured with the next migration type task. If not, it indicates that this is the final step and execution is complete. If there is a next task, it is automatically created and then issued, thus realizing the automated creation and issuance of tasks of different migration types.

[0069] 2. Task creation unit

[0070] The task creation module is the stage for initializing tasks in each module. This stage first creates the task and automatically triggers the task migration preparation stage. The detailed description is as follows:

[0071] 1. System Configuration

[0072] Migrants for each module must first configure which modules need to be migrated for which migration task types. They can also configure custom configurations for each migration module (such as fixed default values) and store them in the configuration center.

[0073] 2. Create and automatically trigger migration preparation

[0074] After receiving a task message from the master control, the system retrieves the corresponding data module based on the master control task information. If no task creation is required (no matching data module is found), the message is ignored. Otherwise, a migration task for the corresponding module (i.e., migration module task) is created. This task involves three steps: preparing details and parameters, executing the migration, and comparing the results. Upon successful creation, migration preparation is automatically triggered and a message is sent to MQ. This approach, through MQ self-production and self-consumption, decouples each unit and helps improve system throughput.

[0075] 3. Task Detail Preparation Unit

[0076] The migration task details preparation module primarily completes task details and prepares interface input parameters. It is a critical step in the entire migration process. Highly accurate migration parameters mean a high migration success rate and can avoid large-scale data manipulation later. Detailed instructions are as follows:

[0077] 1. Start the task preparation phase

[0078] To prevent tasks from being processed multiple times, when the task preparation phase start message is received, a lock (which can be a database lock or a distributed lock) is first acquired. Only after the lock is acquired can the task be executed.

[0079] After acquiring the lock, query the task again. If it is in the Preparing stage or Ready stage, it means that the message is already being processed or has been processed and no further processing is required. Otherwise, update the task to Preparing stage and send a message to the task monitoring and reporting module to inform it that the task has entered the detailed preparation stage. Then, start the migration preparation.

[0080] 2. Construct migration details and corresponding input parameters

[0081] Query which data details need to be migrated in the current module. It may be one, tens of thousands, or even millions. At this time, it is uniformly implemented according to the logic of sharding and multi-threaded parallel processing.

[0082] Each shard corresponds to multiple data details to be migrated. Multi-threading is introduced, and each thread processes one shard. The migration interface corresponding to the module is cyclically queried (there can be multiple ones), and the input parameters of the migration interface are constructed based on the output parameters of each migration interface. The task details and parameters are stored.

[0083] 3. Scheduled task scanning and automatically triggering the next stage

[0084] Start the scheduled task and scan the tasks that have been completed in the preparation phase (the condition is: preparation is completed but migration has not been started). If a task that meets the conditions is scanned, the lock is released, the migration execution of the task is automatically triggered, and the task information is sent to MQ.

[0085] 4. Task Details Execution Migration Unit

[0086] Once the task details and migration parameters are prepared, the migration execution phase begins. This module completes the task execution work in this phase: obtaining the migration parameters and calling the migration interface to complete the data migration. The detailed description is as follows:

[0087] 1. Start the task migration execution phase

[0088] To prevent tasks from being processed multiple times, a lock is first acquired when a message indicating that the task details are started during the migration phase is received. Execution can only be performed after the lock is acquired.

[0089] After acquiring the lock, query the task again. If it is migrating or completed, it means that the message is already being processed or has been processed and no further processing is required. Otherwise, update the task to migrating and send a message to the monitoring module to inform it that the task has entered the detailed migration execution phase. Then, start the migration of the detailed data.

[0090] 2. Call the migration interface to complete data migration

[0091] Query the detailed data that has been prepared by the current module, which may be one, tens of thousands or even millions. At this time, it is uniformly implemented according to the logic of sharding and multi-threaded parallel processing.

[0092] Each shard corresponds to multiple data details to be migrated. Multi-threading is introduced, and each thread processes one shard. The migration interface corresponding to the module is called cyclically (there can be multiple). The success of the migration is determined based on the result returned by the migration interface. At the same time, the parameters returned by the migration interface are saved in the parameter table of the details.

[0093] 3. Scheduled task scanning and automatically triggering the next stage

[0094] Start a scheduled task to scan completed tasks during the migration execution phase (the condition is that the execution is completed but the comparison is not started). If a task that meets the conditions is scanned, the lock is released, the task comparison is automatically triggered, and the task information is sent to MQ.

[0095] 5. Task details comparison unit

[0096] After each detail of the task successfully calls the migration interface, the original system data has been written to the new system. To ensure that the migrated data meets expectations, a comparison is required: the migrated data must be consistent with each field of the pre-migration data. The details are as follows:

[0097] 1. Start the task comparison phase

[0098] To prevent tasks from being processed multiple times, a lock is first acquired when the task details comparison phase start message is received. Only after the lock is acquired can the task be executed.

[0099] After acquiring the lock, query the task again. If it is in comparison or comparison completed, it means that the message is already being processed or has been processed, and no processing is required this time. Otherwise, update the task to comparison, and then send it to the task monitoring and reporting module to inform that the task has entered the comparison stage, and then start the comparison of detailed data.

[0100] 2. Call the query interface to obtain the post-migration data and compare it with the pre-migration data

[0101] Query the migration details of the current module that has been migrated. It may be one, tens of thousands, or even millions of migration details. At this time, it is uniformly implemented according to the logic of sharding and multi-threaded parallel processing.

[0102] Each shard corresponds to multiple data details to be compared. Multi-threading is introduced, and each thread processes one shard. The query interface corresponding to the module is called cyclically (can be multiple) to obtain the migrated data, and then compared with the data before migration. If they are consistent, the comparison is successful. If they are inconsistent, it means that the comparison failed. The input and output parameters of the query during the comparison are saved in the detailed parameter table.

[0103] 3. Scheduled task scanning and automatically triggering migration preparation for the next module

[0104] Start a scheduled task to scan for completed tasks in the migration comparison phase (the condition is that the comparison is completed but not ended). If a task that meets the conditions is scanned, the lock is released and the task is updated to be completed.

[0105] Then determine whether the current task has a task in the next module. If not, end it. If so, automatically trigger the migration preparation of the next module task (send the task to MQ, and the process of the next task enters the task detail preparation stage), thereby completing the automatic flow of a single batch of tasks.

[0106] 6. Task monitoring and reporting unit

[0107] To ensure that tasks at each stage can reach a completion state (full success, partial success, or complete failure) and can also interact with the master control (reporting task status), a task monitoring and reporting module is introduced. This module consumes the messages sent when each stage is started, and then regularly monitors the execution status of each task at each stage. The details are as follows:

[0108] 1. Put the task into the monitoring queue

[0109] Receive the task start monitoring message and put the task into the corresponding queue according to the module and stage it corresponds to. In order to effectively store the task queue, two-layer queues are introduced here: the secondary queue (multiple in number) stores task information, and the length of each queue is controlled at 100 (configurable); the primary queue (can be 1 in number) stores the key value and the number of tasks of each secondary queue and controls the length to 10,000 (configurable). Through these two layers of queues, it can support data volume in the millions (if it needs to be increased, it can be done by adjusting the queue length). Multi-layer cache queues achieve efficient concurrent processing capabilities.

[0110] 2. Regularly scan queue task information and update task status

[0111] Get a certain number of tasks from each queue at a fixed time, then loop through each task to get the detailed execution status of the task (including: the total number of tasks to be executed, success, failure, not executed, and executing). The execution results include:

[0112] Incomplete execution: Not executed + Executing > 0

[0113] Execution completed: Total number of tasks to be executed = success + failure

[0114] Execution completion includes the following three states:

[0115] All executions are successful: Total number of task details to be executed = Success

[0116] Partial execution failure: Total number of tasks to be executed = success + failure

[0117] All executions failed: Total number of tasks to be executed = Failed

[0118] If the execution is not completed, the task is put back into the queue and waits for the next monitoring. Otherwise, the task status is updated according to the execution result (all successful, partial failed, all failed).

[0119] 3. Report task status

[0120] After the task status is updated, the migration result needs to be reported. If not all successful, the migration failure is reported (the report message can indicate which link failed). Otherwise (all successful), it is necessary to determine the link the task is in: if it is the comparison link, the task migration is reported as successful; if not, the task is reported as being executed (the report message can indicate which link succeeded).

[0121] By connecting the above modules in series, the two-layer task-driven migration was completed. Through the automatic triggering of each link, the goal of triggering the overall automated migration with one click was finally achieved, greatly improving the migration efficiency and data accuracy.

[0122] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and maintain the security of user personal information, network security, and national security.

[0123] For ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should be aware that the present invention is not limited to the order of the actions described, and certain steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily required to implement the present invention.

[0124] In order to better implement the above solutions of the embodiments of the present invention, relevant devices for implementing the above solutions are also provided below.

[0125] See also Figure 9 As shown, the device 900 for automatically migrating data provided by an embodiment of the present invention is used to migrate data from the migrating party to the migrating party, and may include: a general control unit 901, a task creation unit 902, a task detail preparation unit 903, a task detail execution migration unit 904, a task detail comparison unit 905 and a task monitoring and reporting unit 906.

[0126] Among them, the general control unit 901 is used to determine the current migration link of the current migration project based on the pre-configured multiple migration links of the current migration project and the execution order between the migration links, and create a migration link task corresponding to the current migration link; the task creation unit 902 is used to use the data modules contained in the pre-configured migration links to create the current migration module task in the migration link task; the task detail preparation unit 903 is used to obtain the detailed data to be migrated of the current migration module task from the migrating party, and store the obtained detailed data to be migrated in a preset database; the task detail execution migration unit 904 is used to write the detailed data to be migrated in the database to the migrating party; the task detail comparison unit 905 is used to perform a comparison of the detailed data to be migrated between the database and the migrating party after the writing is completed; the task monitoring and reporting unit 906 is used to monitor the migration status of the detailed data to be migrated, obtain the execution status of the current migration module task according to the migration status, and report it to the general control unit.

[0127] In an embodiment of the present invention, the task detail preparation unit 903 may be further configured to determine input parameters of the migration interface of the migrating party based on the pre-acquired description information of the detail data to be migrated, and use the input parameters to call the migration interface to obtain the detail data to be migrated.

[0128] In a specific application, the task detail preparation unit 903 can be further used to: obtain the output parameters of the migration-out interface before, after or at the same time as calling the migration-out interface using the input parameters to obtain the detailed data to be migrated; construct the input parameters of the migration-in interface of the migrating party based on the output parameters, and store the input parameters of the migration-in interface in the database.

[0129] In actual applications, the task detail execution migration unit 904 can be further used to: in response to the preset execution migration conditions being met, read the input parameters of the migration interface from the database, and use the input parameters of the migration interface to call the migration interface to write the detail data to be migrated in the database into the migrating party.

[0130] In one embodiment, the task detail comparison unit 905 can be further used to: use the input parameters of the migration interface to call the query interface of the migrating party to obtain the post-migration detail data; call the query interface of the database according to the input parameters of the migration interface to obtain the pre-migration detail data; compare whether the post-migration detail data is consistent with the pre-migration detail data.

[0131] Preferably, the task creation unit 902 can be further used to: after creating the current migration module task, generate a module task creation message and send it to a preset message queue; the module task creation message is used to trigger the acquisition of the detailed data to be migrated in the migrating party; the task detail preparation unit 903 can be further used to: after storing the detailed data to be migrated in the database, generate a preparation completion message and send it to the message queue; the preparation completion message is used to trigger the writing of the detailed data to be migrated; the task detail execution migration unit 904 can be further used to: after writing the detailed data to be migrated to the migrating party, generate a migration completion message and send it to the message queue; the migration completion message is used to trigger the comparison of the detailed data to be migrated; the task detail comparison unit 905 can be further used to: after the comparison is completed, generate a comparison completion message and send it to the message queue.

[0132] As a preferred solution, the task monitoring and reporting unit 906 can be further used to: obtain the module task creation message, the preparation completion message, the migration completion message and the comparison completion message from the message queue, and determine the migration status of the detailed data to be migrated based on the module task creation message, the preparation completion message, the migration completion message and the comparison completion message.

[0133] In an optional technical solution, the task monitoring and reporting unit 906 can be further used to: pre-establish multiple secondary queues for storing the migration status of the detailed data to be migrated and a primary queue for storing the identifier of the secondary queue; in response to updating the migration status of any detailed data to be migrated, obtain the corresponding secondary queue identifier from the primary queue according to the identifier of the detailed data to be migrated, and locate the detailed data to be migrated in the secondary queue indicated by the secondary queue identifier.

[0134] In an optional implementation, the task monitoring and reporting unit 906 may be further used to: when it is monitored that the migration status of any detailed data to be migrated of the current migration module task is execution failure, determine the execution status of the current migration module task as failure; when it is monitored that the migration status of each detailed data to be migrated of the current migration module task is execution success, determine the execution status of the current migration module task as success.

[0135] In addition, in an embodiment of the present invention, the general control unit 901 can be further used to: issue a migration exception reminder when the execution status of the current migration module task is a failure; create the next migration module task of the migration link task and automatically execute it when the execution status of the current migration module task is a success; and determine the next migration link of the current migration project according to the execution order between the migration links, create a migration link task corresponding to the next migration link and automatically execute it when the execution status of each migration module task in the migration link task is a success.

[0136] According to the technical solution of the embodiments of the present invention, an automatic, reusable, high-performance, and highly scalable general data migration framework is provided for actual data migration scenarios. The data migration framework includes a master control unit, a task creation unit, a task detail preparation unit, a task detail execution migration unit, a task detail comparison unit, and a task monitoring and reporting unit. The preparation, execution, and comparison processes of the detailed data to be migrated are automatically executed in a task-driven mode in which the units are completely decoupled from each other. For specific migration projects, only the configuration of each migration link and its execution order and the determination of relevant parameters of the detailed data can be triggered with one click in the master control unit, thereby realizing the automatic creation and efficient execution of migration tasks of various granularities, reducing manual intervention and lowering the labor costs of products, R&D, testing, etc. At the same time, the task execution status of the entire migration process is automatically reported by the task monitoring and reporting unit, which helps to detect anomalies in a timely manner and then resolve them through manual intervention.

[0137] Figure 10 An exemplary system architecture 1000 is shown to which the method or apparatus for automatically migrating data according to an embodiment of the present invention may be applied.

[0138] like Figure 10 As shown, system architecture 1000 may include terminal devices 1001, 1002, and 1003, a network 1004, and a server 1005 (this architecture is merely an example, and the components included in the specific architecture may be adjusted based on the specific application). Network 1004 is a medium for providing a communication link between terminal devices 1001, 1002, and 1003 and server 1005. Network 1004 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0139] Users can use terminal devices 1001, 1002, 1003 to interact with server 1005 via network 1004 to receive or send messages, etc. Various client applications can be installed on terminal devices 1001, 1002, 1003, such as a data migration application (only as an example).

[0140] The terminal devices 1001 , 1002 , and 1003 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0141] Server 1005 may be a server that provides various services, such as a backend server that supports data migration applications operated by users using terminal devices 1001, 1002, and 1003. The backend server may process received data migration requests and provide feedback of processing results (e.g., migration results—for example only) to terminal devices 1001, 1002, and 1003.

[0142] It should be noted that the method for automatically migrating data provided in the embodiment of the present invention can be executed by the server 1005 , and accordingly, the device for automatically migrating data can be set in the server 1005 .

[0143] It should be understood that Figure 10 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0144] The present invention also provides an electronic device. The electronic device in an embodiment of the present invention includes: one or more processors; and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for automatically migrating data provided by the present invention.

[0145] Reference below Figure 11 , which shows a structural diagram of a computer system 1100 of an electronic device suitable for implementing an embodiment of the present invention. Figure 11 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0146] like Figure 11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the computer system 1100 are also stored in the RAM 1103. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0147] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, and the like; an output section 1107 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN card or a modem. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1110 as needed, so that computer programs read therefrom can be installed in the storage section 1108 as needed.

[0148] In particular, according to embodiments disclosed herein, the processes described in the main step diagrams above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via the communication section 1109 and / or installed from removable media 1111. When the computer program is executed by the central processing unit 1101, the above-described functions defined in the system of the present invention are performed.

[0149] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as: a processor comprising: a master control unit, a task creation unit, a task detail preparation unit, etc. The names of these units do not, in certain circumstances, constitute limitations on the units themselves. For example, the master control unit may also be described as "a unit that provides migration link tasks to the task creation unit."

[0152] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the device performs the following steps: determining the current migration link of the current migration project according to the multiple migration links of the pre-configured current migration project and the execution order between the migration links, and creating a migration link task corresponding to the current migration link; creating the current migration module task in the migration link task using the data modules included in the pre-configured migration links; obtaining the detailed data to be migrated of the current migration module task from the migrating party, and storing the obtained detailed data to be migrated in a preset database; writing the detailed data to be migrated in the database to the migrating party; after the writing is completed, performing a comparison of the detailed data to be migrated between the database and the migrating party; monitoring the migration status of the detailed data to be migrated, and obtaining the execution status of the current migration module task according to the migration status.

[0153] In the technical solution of the embodiment of the present invention, an automatic, reusable, high-performance, and highly scalable general data migration framework is provided for actual data migration scenarios. The data migration framework includes a master control unit, a task creation unit, a task detail preparation unit, a task detail execution migration unit, a task detail comparison unit, and a task monitoring and reporting unit. The preparation, execution, and comparison processes of the detailed data to be migrated are automatically executed in a task-driven mode with complete decoupling between the units. For specific migration projects, it is only necessary to configure the migration links and their execution order and determine the relevant parameters of the detailed data, which can be triggered with one click in the master control unit, thereby realizing the automatic creation and efficient execution of migration tasks of various granularities, reducing manual intervention and lowering the labor costs of products, R&D, testing, etc. At the same time, the task execution status of the entire migration process is automatically reported by the task monitoring and reporting unit, which helps to detect anomalies in a timely manner and then resolve them through manual intervention.

[0154] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for automatically migrating data, characterized in that: include: Determining a current migration link of the current migration project according to a plurality of pre-configured migration links of the current migration project and an execution order between the migration links, and creating a migration link task corresponding to the current migration link; Using the pre-configured data modules included in each migration link to create a current migration module task in the migration link task; Obtaining detailed data to be migrated of the current migration module task from the migrating party, storing the obtained detailed data to be migrated in a preset database; and writing the detailed data to be migrated in the database to the migrating party; After writing is completed, a comparison of the detailed data to be migrated is performed between the database and the migration destination; The migration status of the detailed data to be migrated is monitored, and the execution status of the current migration module task is obtained according to the migration status.

2. The method according to claim 1, characterized in that The obtaining of the detailed data to be migrated of the current migration module task from the migrating party includes: The input parameters of the migration interface of the migrating party are determined according to the pre-acquired description information of the detailed data to be migrated, and the migration interface is called using the input parameters to obtain the detailed data to be migrated.

3. The method according to claim 2, characterized in that The method further comprises: Before, after, or simultaneously with calling the migration interface using the input parameters to obtain the detailed data to be migrated, obtaining output parameters of the migration interface; The input parameters of the migration interface of the migrating party are constructed according to the output parameters, and the input parameters of the migration interface are stored in the database.

4. The method according to claim 3, characterized in that Writing the detailed data to be migrated in the database to the migration-in party includes: In response to the preset migration execution condition being met, the input parameters of the migration-in interface are read from the database, and the migration-in interface is called using the input parameters of the migration-in interface to write the detailed data to be migrated in the database into the migrating party.

5. The method according to claim 3, characterized in that The comparing of the detailed data to be migrated between the database and the migrating party includes: Using the input parameters of the migration interface, the query interface of the migration party is called to obtain the detailed data after migration; according to the input parameters of the migration interface, the query interface of the database is called to obtain the detailed data before migration; Compare the detailed data after migration with the detailed data before migration to see if they are consistent.

6. The method according to claim 1, characterized in that The method further comprises: After creating the current migration module task, a module task creation message is generated and sent to a preset message queue; the module task creation message is used to trigger the acquisition of the detailed data to be migrated in the migrating party; After storing the detailed data to be migrated in the database, generating a preparation completion message and sending it to the message queue; the preparation completion message is used to trigger the writing of the detailed data to be migrated; After the detailed data to be migrated is written into the migration destination, a migration completion message is generated and sent to the message queue; the migration completion message is used to trigger the comparison of the detailed data to be migrated; After the comparison is completed, a comparison completion message is generated and sent to the message queue.

7. The method according to claim 6, characterized in that The monitoring of the migration status of the detailed data to be migrated includes: The module task creation message, the preparation completion message, the migration completion message and the comparison completion message are obtained from the message queue, and the migration status of the detailed data to be migrated is determined based on the module task creation message, the preparation completion message, the migration completion message and the comparison completion message.

8. The method according to claim 1, characterized in that The method further comprises: Pre-establishing a plurality of secondary queues for storing the migration status of the detailed data to be migrated and a primary queue for storing the identifiers of the secondary queues; In response to updating the migration status of any detailed data to be migrated, a corresponding secondary queue identifier is obtained from the primary queue according to the identifier of the detailed data to be migrated, and the detailed data to be migrated is located in the secondary queue indicated by the secondary queue identifier.

9. The method according to claim 1, characterized in that The obtaining, according to the migration status, the execution status of the current migration module task includes: When it is monitored that the migration status of any to-be-migrated detailed data of the current migration module task is execution failure, determining the execution status of the current migration module task as failure; When it is monitored that the migration status of each to-be-migrated detailed data of the current migration module task is successfully executed, the execution status of the current migration module task is determined to be successful.

10. The method according to claim 9, characterized in that The method further comprises: If the execution status of the current migration module task fails, a migration exception reminder is issued; If the execution of the current migration module task is successful, creating the next migration module task of the migration link task and automatically executing it; When the execution of each migration module task in the migration link task is successful, the next migration link of the current migration project is determined according to the execution order between the migration links, and the migration link task corresponding to the next migration link is created and automatically executed.

11. A device for automatically migrating data, characterized in that: include: A master control unit is configured to determine a current migration link of the current migration project according to a plurality of pre-configured migration links of the current migration project and an execution order of the migration links, and to create a migration link task corresponding to the current migration link; A task creation unit, configured to create a current migration module task in the migration link task by using the pre-configured data modules included in each migration link; a task detail preparation unit, configured to obtain the to-be-migrated detailed data of the current migration module task from the migrating party, and store the obtained to-be-migrated detailed data in a preset database; A task detail execution migration unit, configured to write the detail data to be migrated in the database into the migration destination; a task detail comparison unit, configured to compare the detailed data to be migrated between the database and the migrating party after writing is completed; The task monitoring and reporting unit is used to monitor the migration status of the detailed data to be migrated, obtain the execution status of the current migration module task according to the migration status, and report it to the general control unit.

12. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images