Data migration method and apparatus

Abstract

The present disclosure provides a data migration method and device, the method comprising: in response to the target database being in an initialization state, determining whether a target instance running a target scheme has completed initialization; in response to the target instance not having completed initialization, reading scheme configuration information of the target scheme; generating a to-be-processed task queue of the target scheme according to the scheme configuration information to complete initialization of the target instance; and using the target instance to process a plurality of to-be-processed timing tasks to implement data migration of to-be-migrated data. Thus, the data migration process is divided into a plurality of to-be-processed timing tasks, which are processed in order in the to-be-processed task queue, thereby avoiding a problem of excessive data volume involved in migration, and automatically and orderly completing data migration of to-be-migrated data. This not only improves the efficiency and accuracy of data migration, but also reduces the possibility of human intervention and errors.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data migration method and apparatus. Background Technology

[0002] As the system goes online for longer, more and more data will accumulate in the system, which may cause performance problems. Therefore, it is necessary to migrate some historical data in the system.

[0003] Currently, data migration is usually performed manually by writing scripts on a regular basis. However, since the manual scheduling method is quite cumbersome, the interval between two data migrations is usually long. The longer the interval between two data migrations, the larger the amount of data involved in the migration, resulting in a longer data migration time each time. Furthermore, the data migration efficiency of the manual scheduling method is low, which may affect the normal use of system data. Summary of the Invention

[0004] This disclosure provides a data migration method and apparatus to at least partially solve one of the technical problems in the related art. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a data migration method is provided, comprising: in response to a target database being in an initialization state, determining whether a target instance running a target scheme has completed initialization; wherein the target scheme is used to migrate data to be migrated in the target database; in response to the target instance not completing initialization, reading scheme configuration information of the target scheme; generating a task queue to be processed for the target scheme according to the scheme configuration information to complete the initialization of the target instance; wherein the task queue includes multiple timed tasks to be processed; and using the target instance to process the multiple timed tasks to achieve the data migration of the data to be migrated.

[0006] According to a second aspect of the present disclosure, a data migration apparatus is provided, comprising: a judging module, configured to judge whether a target instance running a target scheme has completed initialization in response to a target database being in an initialization state; wherein the target scheme is used to migrate data to be migrated in the target database; a reading module, configured to read scheme configuration information of the target scheme in response to the target instance not completing initialization; a generating module, configured to generate a queue of tasks to be processed for the target scheme according to the scheme configuration information to complete the initialization of the target instance; wherein the task queue includes a plurality of scheduled tasks to be processed; and a processing module, configured to use the target instance to process the plurality of scheduled tasks to be processed to realize the data migration of the data to be migrated.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data migration method as described in the first aspect of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a data migration method as described in the first aspect of the present disclosure.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the data migration method as described in the first aspect of the present disclosure.

[0010] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0011] In this technical solution, in response to the target database being in an initialization state, it is determined whether the target instance running the target scheme has completed initialization; wherein, the target scheme is used to migrate the data to be migrated in the target database; in response to the target instance not completing initialization, the scheme configuration information of the target scheme is read; according to the scheme configuration information, a task queue to be processed for the target scheme is generated to complete the initialization of the target instance; wherein, the task queue includes multiple scheduled tasks to be processed; the target instance is used to process the multiple scheduled tasks to be processed to realize the data migration of the data to be migrated; thus, by checking the initialization status of the target instance, and generating a task queue to be processed for the target scheme according to the scheme configuration information of the target scheme when the target instance has not completed initialization, the target instance is made to be in a correct and ready state before data migration, and the data migration process is decomposed into multiple scheduled tasks to be processed, which are processed step by step in the order in the task queue, so that even if the interval between two data migrations is long, the migration will not be affected. This addresses the issue of excessively large data volumes and enables automatic and orderly data migration, improving efficiency and accuracy while reducing the possibility of human intervention and errors. Specifically, before executing each process node of the target solution, it checks whether user control instructions have been received, enhancing the controllability of the data migration. If no control instructions are received, it accurately retrieves the corresponding scheduled task from multiple pending scheduled tasks, ensuring targeted and accurate data migration and avoiding unnecessary resource waste. Next, it determines whether the current process node is a batch-committed transaction node, facilitating the adoption of appropriate processing strategies based on different node types, thus improving data migration efficiency. Furthermore, when the current process node is a batch-committed transaction node, it uses the target instance for at least one batch processing, achieving batch processing and high efficiency. This allows for the rapid and accurate processing of large amounts of data, improving reliability and efficiency, and enhancing the controllability and flexibility of the entire migration process.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0014] Figure 1 This is a flowchart illustrating the data migration method shown in the first embodiment of this disclosure;

[0015] Figure 2 This is a flowchart illustrating the data migration method shown in the second embodiment of this disclosure;

[0016] Figure 3 This is a flowchart illustrating the data migration method shown in the third embodiment of this disclosure;

[0017] Figure 4 This is a schematic flowchart of the data migration method shown in the fourth embodiment of this disclosure;

[0018] Figure 5 This is a schematic diagram of the process for initializing the current instance as shown in the embodiments of this disclosure;

[0019] Figure 6 This is a schematic diagram of the creation process of a scheduled task to be processed, as shown in an embodiment of this disclosure.

[0020] Figure 7 This is a schematic diagram of the processing flow of the target solution shown in the embodiments of this disclosure;

[0021] Figure 8 This is a schematic diagram of the structure of the data migration apparatus shown in the fifth embodiment of this disclosure;

[0022] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0024] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0025] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0026] The data migration method and apparatus of embodiments of this disclosure are described below with reference to the accompanying drawings.

[0027] Figure 1 This is a flowchart illustrating the data migration method shown in the first embodiment of this disclosure.

[0028] like Figure 1 As shown, this data migration method includes the following steps:

[0029] Step 101: In response to the target database being in the initialization state, determine whether the target instance running the target scheme has completed initialization.

[0030] The target scheme is used to migrate the data to be migrated in the target database.

[0031] To avoid unnecessary waste of resources and time, one possible approach is to quickly determine whether the initialization actions of the target instance need to be performed by querying the status of the target database (e.g., REDIS) to run the target solution.

[0032] As an example, when the target database is in the initialization state, it is determined whether the target instance running the target scheme has completed initialization, where the target scheme is used to migrate the data to be migrated in the target database.

[0033] As another example, if the target database is not in the initialization state, such as when the target database is in the completed state, it means that the relevant data migration or solution process configuration has been completed, and there is no need to initialize the target instance to run the target solution.

[0034] Step 102: In response to the target instance not completing initialization, read the scheme configuration information of the target scheme.

[0035] To improve the flexibility of data migration by adapting it to actual user needs, as an example, the scheme configuration information of the target solution is read from the target database when the target instance has not been initialized. It should be noted that the scheme configuration information is obtained by the user through configuration on the visual configuration page according to their needs.

[0036] Step 103: Based on the scheme configuration information, generate a queue of tasks to be processed for the target scheme to complete the initialization of the target instance.

[0037] The task queue contains multiple scheduled tasks awaiting processing.

[0038] To improve the accuracy and efficiency of data migration, as an example, the scheme configuration information is read and parsed, and then multiple pending scheduled tasks are created based on the scheme configuration information and written into the pending task queue.

[0039] It should be noted that, in order to ensure that the initialization of the target instance is re-executed when the scheme configuration information is updated, the status of the target database is updated from the initialization state to the completion state when the target instance initialization is completed, and the status of the target database is updated from the completion state to the initialization state when the scheme configuration information is updated.

[0040] Step 104: Using the target instance, process multiple scheduled tasks to achieve data migration of the data to be migrated.

[0041] Furthermore, based on the target instance, multiple pending scheduled tasks are automatically executed. These tasks perform data migration operations according to a predetermined schedule and rules, thereby achieving efficient and accurate migration of the data to be migrated from the source to the target.

[0042] In summary, by responding to the target database being in an initialization state, it is determined whether the target instance running the target scheme has completed initialization; wherein, the target scheme is used to migrate the data to be migrated in the target database; in response to the target instance not completing initialization, the scheme configuration information of the target scheme is read; according to the scheme configuration information, a task queue to be processed for the target scheme is generated to complete the initialization of the target instance; wherein, the task queue includes multiple scheduled tasks to be processed; the target instance is used to process the multiple scheduled tasks to be processed to realize the data migration of the data to be migrated; thus, by checking the initialization status of the target instance, and in response to the target instance not completing initialization, it is determined whether the target instance has completed initialization. In this scenario, based on the configuration information of the target solution, a queue of tasks to be processed for the target solution is generated. This ensures that the target instance is in a correct and ready state before data migration. Furthermore, the data migration process is decomposed into multiple scheduled tasks to be processed. These scheduled tasks are processed step by step in the order they are in the queue. This prevents the problem of excessive data volume during migration even if the interval between two data migrations is long. At the same time, it achieves automatic and orderly completion of the data migration. This automated processing method based on task queues not only improves the efficiency and accuracy of data migration but also reduces the possibility of human intervention and errors.

[0043] To clearly illustrate how the target instance is used to process multiple pending timed tasks in the above embodiments, this disclosure proposes another data migration method.

[0044] Figure 2 This is a schematic flowchart of the data migration method shown in the second embodiment of this disclosure.

[0045] like Figure 2 As shown, this data migration method includes the following steps:

[0046] Step 201: In response to the target database being in the initialization state, determine whether the target instance running the target scheme has completed initialization.

[0047] The target scheme is used to migrate the data to be migrated in the target database.

[0048] Step 202: In response to the target instance not completing initialization, read the scheme configuration information of the target scheme.

[0049] Step 203: Based on the scheme configuration information, generate a queue of tasks to be processed for the target scheme to complete the initialization of the target instance.

[0050] The task queue contains multiple scheduled tasks awaiting processing.

[0051] Step 204: In response to not receiving a scheme control instruction, determine the current process node from at least one process node of the target scheme.

[0052] Among them, the scheme control command is used to terminate or pause the current process node.

[0053] To improve the controllability of data migration, as a possible implementation, before executing each process node of the target scheme, it is determined whether a scheme control instruction from the user has been received. The scheme control instruction is used to terminate or pause the current process node.

[0054] As an example, when no scheme control instruction is received, the current process node to be executed is determined from at least one process node of the target scheme, and then the scheduled task to be processed corresponding to the current process node is executed.

[0055] As another example, upon receiving a scheme control instruction, the execution status of the current process node and the execution status of the target scheme are updated, and the current process node is paused or terminated.

[0056] Step 205: Obtain the scheduled task corresponding to the current process node from multiple scheduled tasks to be processed.

[0057] To ensure that the data migration process proceeds according to the predetermined sequence and logic, and to avoid confusion and conflict between tasks, as an example, each process node corresponds to at least one scheduled task to be processed. From multiple scheduled tasks to be processed, the scheduled task to be processed that matches the current process node is identified and extracted.

[0058] Step 206: Determine whether the current process node in the target solution is a process node for batch submission transactions.

[0059] It is important to understand that each process node in the target solution has a corresponding flag bit. The value of this flag bit is used to indicate whether the corresponding process node is a process node for batch submission transactions.

[0060] As an example, the value of the flag bit of the current process node in the target solution can be used to determine whether the current process node is a process node of batch commit transactions. For example, a value of "true" indicates that the current process node is a process node of batch commit transactions, and a value of "false" indicates that the current process node is a process node of non-batch commit transactions.

[0061] Step 207: In response to the current process node being a process node for batch commit transactions, use the target instance to process at least one batch of the data to be migrated as indicated by the scheduled task to be processed corresponding to the current process node.

[0062] As an example, when the current process node is a process node for batch submission transactions, the target instance is used to process the data to be migrated indicated by the scheduled task to be processed corresponding to the current process node in at least one batch.

[0063] At least one batch processing step includes the following steps:

[0064] 1. Determine whether a scheme control instruction for one of the batch processing methods has been received; i is a positive integer;

[0065] 2. If not, then retrieve one of the batches of data to be migrated from the data to be migrated;

[0066] 3. Using the target instance, perform batch processing on one of the batches of data to be migrated.

[0067] In other words, to further improve the controllability of the data migration process, for the first batch processing, it is determined whether a scheme control instruction has been received. If no scheme control instruction is received, the data to be migrated in the first batch is determined from the data to be migrated, and the target instance is used to process the data to be migrated in the first batch. At the end of the first batch processing, it is determined whether a scheme control instruction has been received. If no scheme control instruction is received, the data to be migrated in the second batch is obtained from the data to be migrated, and the target instance is used to process the data to be migrated in the second batch. This process continues until the last batch of at least one batch processing is completed. For each batch processing, if a scheme control instruction is received, the data migration is paused or terminated, and the execution status of the current process node to which the batch belongs and the execution status of the target scheme are updated.

[0068] To facilitate monitoring the execution progress of data migration, as an example, in response to the completion of the last batch in at least one batch processing, it is determined whether the current process node is the last process node in the target solution; if the current process node is the last process node, the execution status of the current process node and the execution status of the target solution are updated; if the current process node is not the last process node, the execution status of the current process node is updated; in addition, each time a solution control command is received, the execution status of the process node corresponding to the received solution control command and the execution status of the target solution are updated.

[0069] As another example, in response to the current process node being a process node that is not a batch commit transaction, the processing logic of the pending scheduled task corresponding to the current process node is obtained, and the target instance is used to process the data to be migrated indicated by the pending scheduled task corresponding to the current process node according to the processing logic.

[0070] In other words, when the current process node is a process node that is not a batch-committed transaction, the scheduled task to be processed corresponding to the current process node has predefined processing logic. Therefore, by using the target instance, according to the predefined processing logic, the data to be migrated indicated by the scheduled task to be processed corresponding to the current process node is processed accurately and efficiently. This process enables data migration to both conform to business logic and be performed efficiently in non-batch mode.

[0071] It should be noted that the execution process of steps 201 to 203 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0072] In summary, in response to the absence of a scheme control instruction, the current process node is determined from at least one process node in the target scheme; the pending scheduled task corresponding to the current process node is obtained from multiple pending scheduled tasks; it is determined whether the current process node in the target scheme is a process node for batch commit transactions; in response to the current process node being a process node for batch commit transactions, the target instance is used to perform at least one batch processing on the data to be migrated indicated by the pending scheduled task corresponding to the current process node. Thus, by determining whether a scheme control instruction from the user has been received before executing each process node of the target scheme, the controllability of data migration is improved. Furthermore, when no scheme control instruction is received, the pending scheduled task corresponding to the current process node is accurately obtained from multiple pending scheduled tasks, achieving targeted and accurate data migration and avoiding unnecessary resource waste. Next, determining whether the current process node is a batch commit transaction node helps to adopt corresponding processing strategies based on different types of process nodes, improving data migration efficiency. Finally, when the current process node is a batch commit transaction node, at least one batch is processed using the target instance, achieving batch processing and high efficiency in data migration. This enables large amounts of data to be processed quickly and accurately, while improving the reliability and efficiency of data migration, as well as enhancing the controllability and flexibility of the entire migration process.

[0073] To clearly illustrate how the target scheme's task queue is generated based on the scheme configuration information in the above embodiments, this disclosure proposes another data migration method.

[0074] Figure 3 This is a flowchart illustrating the data migration method shown in the third embodiment of this disclosure.

[0075] like Figure 3 As shown, this data migration method includes the following steps:

[0076] Step 301: In response to the target database being in the initialization state, determine whether the target instance running the target scheme has completed initialization.

[0077] The target scheme is used to migrate the data to be migrated in the target database.

[0078] Step 302: In response to the target instance not completing initialization, read the scheme configuration information of the target scheme.

[0079] Step 303: In response to acquiring the database lock of the target database, determine whether multiple pending scheduled tasks have been created.

[0080] To ensure data consistency and integrity and prevent data conflicts or inconsistencies during task creation and processing, as an example, a database lock on the target database is acquired. Once the database lock on the target database is acquired, it is determined whether multiple scheduled tasks to be processed have been created. That is, while the target database is locked, it is determined whether the scheduled tasks have been created.

[0081] Step 304: In response to the fact that multiple pending scheduled tasks have not been created, according to the scheme configuration information, the pending scheduled tasks for multiple process nodes of the target scheme are created in sequence and written into the pending task queue.

[0082] Each process node corresponds to at least one scheduled task to be processed.

[0083] To reduce human intervention and errors, as a possible implementation, if multiple pending scheduled tasks have not been created, a queue of pending scheduled tasks can be automatically generated based on the scheme configuration information.

[0084] As an example, based on the scheme configuration information, the pending timed tasks for multiple process nodes of the target scheme are created sequentially. When the pending timed tasks for all process nodes are created, the pending timed tasks for each process node are written into the pending task queue in sequence.

[0085] As another example, based on the scheme configuration information, the scheduled tasks to be processed for each process node of the target scheme are created sequentially. When the scheduled task to be processed for each process node is created, the scheduled task to be processed is written into the task queue.

[0086] Step 305: Using the target instance, process multiple scheduled tasks to achieve data migration of the data to be migrated.

[0087] It should be noted that the execution process of steps 301 to 302 and step 305 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit the implementation of these steps and will not elaborate further.

[0088] In summary, by acquiring a database lock on the target database and determining whether multiple pending scheduled tasks have been created, the database locking mechanism avoids conflicts or data inconsistencies caused by multiple migration tasks operating on the same data resource simultaneously. Furthermore, checking the status of pending scheduled tasks while the database is locked allows for timely detection and handling of issues during task creation, ensuring effective data migration and improving its reliability and stability. Moreover, even when multiple pending scheduled tasks have not been created, the system automatically creates them based on the configuration information, reducing human intervention and errors. Adding pending scheduled tasks to a queue ensures ordered task execution, avoiding resource conflicts and redundant processing, and improving processing efficiency.

[0089] To ensure the orderly execution of tasks during data migration and avoid resource conflicts and duplicate processing, as an example, based on the scheme configuration information, pending scheduled tasks for each process node of the target scheme are created sequentially. Upon completion of the creation of a pending scheduled task at each process node, it is written to the pending task queue. Then, the target instance is used to process multiple pending scheduled tasks to achieve the data migration of the data to be migrated. The following section will combine... Figure 4 The embodiments are described in detail.

[0090] Figure 4 This is a schematic flowchart of the data migration method shown in the fourth embodiment of this disclosure.

[0091] like Figure 4 As shown, this data migration method includes the following steps:

[0092] Step 401: In response to the target database being in the initialization state, determine whether the target instance running the target scheme has completed initialization.

[0093] The target scheme is used to migrate the data to be migrated in the target database.

[0094] Step 402: In response to the target instance not completing initialization, read the scheme configuration information of the target scheme.

[0095] Step 403: In response to acquiring the database lock of the target database, determine whether multiple pending scheduled tasks have been created.

[0096] Step 404: In response to the fact that multiple pending scheduled tasks have not been created, create the pending scheduled tasks for the current process node according to the scheme configuration information.

[0097] To ensure that data is migrated according to user needs, as a possible approach, when multiple pending scheduled tasks have not been created, a pending scheduled task for the current process node can be created based on the scheme configuration information.

[0098] Step 405: In response to the completion of the creation of the pending timed task for the current process node, write the pending timed task for the current process node into the pending task queue.

[0099] To avoid conflicts between tasks, the pending scheduled tasks of the current process node are further written into the pending task queue when the pending scheduled tasks of the current process node are created.

[0100] Step 406: Determine whether the current process node is the last process node in the target solution.

[0101] To avoid unnecessary operations and reduce resource waste, as an example, we determine whether the current process node is the last process node in the target solution. The target solution includes m process nodes.

[0102] Step 407: If yes, then end the writing process of the pending task queue.

[0103] As an example, when the current process node is the last process node in the target solution, that is, when all the pending timers for all process nodes in the target solution have been created and written to the pending task queue, the writing process of the pending task queue ends.

[0104] Step 408: If not, then according to the scheme configuration information, create the pending timed task for the next process node of the current process node, so that when the pending timed task for the next process node is created, the pending timed task for the next process node is written into the pending task queue.

[0105] As another example, when the current process node is not the last process node in the target solution, continue to create the pending timed task for the next process node according to the solution configuration information, and write the pending timed task for the next process node into the pending task queue when the pending timed task for the next process node is created.

[0106] To avoid occupying database resources for extended periods and reduce the waiting time for other threads or processes, as an example, the database lock can be released when multiple pending scheduled tasks are written to the pending task queue, or when multiple pending scheduled tasks have completed creation.

[0107] Step 409: Using the target instance, process multiple scheduled tasks to achieve data migration of the data to be migrated.

[0108] It should be noted that the execution process of steps 401 to 403 and step 409 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit the implementation of these steps and will not elaborate further.

[0109] In summary, by creating the scheduled task for the current process node based on the scheme configuration information when multiple scheduled tasks have not yet been created, the system achieves orderly and accurate task creation, avoiding task omissions or duplicate creation. After the scheduled task for the current process node is created, it is written to the scheduled task queue, achieving orderly queuing and efficient scheduling. The queue mechanism ensures that scheduled tasks are processed according to a predetermined order and priority, reducing waiting time and conflicts between them. Furthermore, by determining whether the current process node is the last process node in the target scheme, the system can accurately control the writing process to the scheduled task queue. If the current process node is the last node, the writing process ends, avoiding unnecessary resource consumption and waste. If the current process node is not the last process node, the system continues to create the scheduled task for the next process node based on the scheme configuration information and writes it to the scheduled task queue, ensuring the entire process proceeds according to the predetermined order and logic, thus improving the integrity of the scheduled task queue.

[0110] Based on any embodiment of this disclosure, the data migration method of this disclosure can also be implemented based on the following steps:

[0111] 1. Upon startup, a looping scheduled task will be initialized to perform instance initialization actions. The initialization flowchart is as follows: Figure 5 As shown, the specific steps include the following:

[0112] 1) Retrieve the initialization status of REDIS (target database). If the current instance (target instance) is not initialized, start the subsequent initialization actions; otherwise, exit directly.

[0113] 2) Read the configuration information (scheme configuration information) of the solution process from the database;

[0114] 3) Loop through configuration information and create scheduled tasks for the solution process;

[0115] 4) After all process configurations are created, update the initialization status to complete, that is, update the target database status from the initialization status to the complete status;

[0116] 5) If the front-end adjusts the solution process configuration information, clear the initialization status on REDIS, that is, update the status of the target database from the completed status to the initialization status;

[0117] 2. The core logic of the scheduled task for creating a solution process is only responsible for generating the task queue of the process involved in the current solution. The generated task queue is then consumed and processed by other background tasks.

[0118] The flowchart for processing scheduled tasks in the loop configuration information creation process is as follows: Figure 6 As shown, the specific steps include the following:

[0119] 1) Attempt to acquire the lock. If the lock is acquired successfully, continue with the subsequent actions; otherwise, exit.

[0120] 2) After acquiring the lock, check if a task has already been created. If it has, exit directly after unlocking.

[0121] 3) If not created, create the queue of tasks to be processed corresponding to the current solution based on the configuration information of the current solution process;

[0122] 4) After creation, release the lock and exit the processing logic;

[0123] 3. After receiving the consumption plan task, the consumer shall, according to... Figure 7 The process shown executes each step, and the specific steps are as follows:

[0124] 1) Before executing the solution steps, first determine whether the current process node is paused or terminated. If so, update the log status and exit the logic directly; otherwise, continue with the subsequent logic.

[0125] 2) If the current process node is a process node that is not a batch commit transaction, then start the transaction before execution, and then execute step by step in sequence until the last step is completed. Then update the log status and end the solution processing logic.

[0126] 3) If the current process node is a process node for batch commit transactions, when a step is retrieved, it first checks if it is a step that initiates a batch transaction. If so, it checks if the process is paused or terminated. If not, it starts the transaction, executes the step logic that requires batch commit, and then commits the batch transaction. After the transaction is committed, it checks if there are any unprocessed batches. If there are, it continues to process the next batch.

[0127] Corresponding to the data migration method provided in the above embodiments, this disclosure also provides a data migration apparatus. Since the data migration apparatus provided in this disclosure corresponds to the data migration method provided in the above embodiments, the implementation of the data migration method is also applicable to the data migration apparatus provided in this disclosure, and will not be described in detail in this disclosure.

[0128] Figure 8This is a schematic diagram of the structure of the data migration apparatus shown in the fifth embodiment of this disclosure.

[0129] like Figure 8 As shown, the data migration device 800 includes: a judgment module 810, a reading module 820, a generation module 830, and a processing module 840.

[0130] The system includes a judgment module 810, which determines whether the target instance running the target scheme has completed initialization in response to the target database being in an initialization state; wherein the target scheme is used to migrate the data to be migrated in the target database; a reading module 820, which reads the scheme configuration information of the target scheme in response to the target instance not completing initialization; a generation module 830, which generates a queue of tasks to be processed for the target scheme based on the scheme configuration information to complete the initialization of the target instance; wherein the task queue includes multiple scheduled tasks to be processed; and a processing module 840, which uses the target instance to process the multiple scheduled tasks to be processed to achieve the data migration of the data to be migrated.

[0131] As one possible implementation of this disclosure, the processing module 840 is configured to, in response to not receiving a scheme control instruction, determine the current process node from at least one process node of the target scheme; wherein the scheme control instruction is used to terminate or pause the current process node; obtain the pending timed task corresponding to the current process node from a plurality of pending timed tasks; determine whether the current process node in the target scheme is a process node for batch submission transactions; and, in response to the current process node being a process node for batch submission transactions, use the target instance to perform at least one batch processing on the data to be migrated indicated by the pending timed task corresponding to the current process node.

[0132] As one possible implementation of this disclosure, at least one batch processing includes: determining whether a scheme control instruction for one batch processing has been received; if not, obtaining the data to be migrated from the data to be migrated for one batch processing; and using a target instance to perform batch processing on the data to be migrated for one batch processing.

[0133] As one possible implementation of this disclosure, the processing module 840 is further configured to, in response to the completion of the last batch processing in at least one batch processing, determine whether the current process node is the last process node in the target scheme; in response to the current process node being the last process node, update the execution status of the current process node and the execution status of the target scheme; and in response to the current process node not being the last process node, update the execution status of the current process node.

[0134] As one possible implementation of this disclosure, the processing module 840 is further configured to, in response to the current process node being a process node of a non-batch submission transaction, obtain the processing logic of the scheduled task to be processed corresponding to the current process node; and, using the target instance, process the data to be migrated indicated by the scheduled task to be processed corresponding to the current process node according to the processing logic.

[0135] As one possible implementation of this disclosure, the generation module 830 is used to determine whether multiple pending timed tasks have been created in response to acquiring the database lock of the target database; in response to the failure to create multiple pending timed tasks, to sequentially create pending timed tasks for multiple process nodes of the target scheme according to the scheme configuration information, and write them into the pending task queue; wherein, each process node corresponds to at least one pending timed task.

[0136] As one possible implementation of this disclosure, the target scheme includes m process nodes and a generation module 830, which is used to: respond to multiple pending timed tasks not being created, create pending timed tasks for the current process node according to scheme configuration information; respond to the completion of the creation of pending timed tasks for the current process node, write the pending timed tasks for the current process node into a pending task queue; determine whether the current process node is the last process node in the target scheme; if yes, end the writing process of the pending task queue; if no, create pending timed tasks for the next process node according to the scheme configuration information, so that when the pending timed tasks for the next process node are created, write the pending timed tasks for the next process node into the pending task queue.

[0137] As one possible implementation of this disclosure, the data migration apparatus 800 further includes a release module.

[0138] The release module is used to release the database lock in response to multiple pending scheduled tasks being written into the pending task queue; or, in response to multiple pending scheduled tasks being created, to release the database lock.

[0139] As one possible implementation of this disclosure, the data migration apparatus 800 further includes an update module.

[0140] The update module is used to update the status of the target database from the initialization state to the completion state; in response to an update of the scheme configuration information, the status of the target database is updated from the completion state to the initialization state.

[0141] The data migration apparatus of this disclosure determines whether the target instance running the target scheme has completed initialization in response to the target database being in an initialization state. The target scheme is used to migrate the data to be migrated from the target database. In response to the target instance not completing initialization, the scheme configuration information of the target scheme is read. Based on the scheme configuration information, a task queue to be processed for the target scheme is generated to complete the initialization of the target instance. The task queue includes multiple scheduled tasks to be processed. The target instance is used to process these scheduled tasks to achieve the data migration of the data to be migrated. Therefore, by checking the initialization state of the target instance and generating a task queue to be processed for the target scheme based on the scheme configuration information when the target instance has not completed initialization, the target instance is ensured to be in a correct and ready state before data migration. Furthermore, the data migration process is decomposed into multiple scheduled tasks to be processed, which are processed sequentially in the task queue. This prevents excessive data volume during migration even with long intervals between two data migrations, and achieves automatic and orderly completion of the data migration. This not only improves the efficiency and accuracy of data migration but also reduces the possibility of human intervention and errors.

[0142] In an exemplary embodiment, an electronic device is also proposed.

[0143] The electronic devices include:

[0144] processor;

[0145] Memory used to store processor-executable instructions;

[0146] The processor is configured to execute instructions to implement the data migration method as proposed in any of the foregoing embodiments.

[0147] As an example, Figure 9 This is a schematic diagram of the structure of an electronic device 900 as shown in an exemplary embodiment of this disclosure, as follows: Figure 9 As shown, the aforementioned electronic device 900 may further include:

[0148] The system includes a memory 910 and a processor 920, and a bus 930 connecting different components (including the memory 910 and the processor 920). The memory 910 stores a computer program, and when the processor 920 executes the program, it implements the data migration method described in the embodiments of this disclosure.

[0149] Bus 930 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0150] Electronic device 900 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 900, including volatile and non-volatile media, removable and non-removable media.

[0151] The memory 910 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 940 and / or cache memory 950. The server 900 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 960 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 9 Not shown; usually referred to as a "hard drive"). Although Figure 9 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 930 via one or more data media interfaces. Memory 910 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0152] A program / utility 980 having a set (at least one) of program modules 970 may be stored, for example, in memory 910. Such program modules 970 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 970 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0153] Electronic device 900 can also communicate with one or more external devices 990 (e.g., keyboard, pointing device, display 991, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 992. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 993. As shown, network adapter 993 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0154] The processor 920 performs various functional applications and data processing by running programs stored in the memory 910.

[0155] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the data migration method of the present disclosure embodiment, and will not be repeated here.

[0156] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to complete the data migration method proposed in any of the above embodiments. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0157] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the data migration method proposed in any of the above embodiments.

[0158] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0159] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data migration method, characterized in that, include: In response to the target database being in an initialization state, it is determined whether the target instance running the target scheme has completed initialization; wherein, the target scheme is used to migrate the data to be migrated in the target database; In response to the target instance not completing initialization, read the scheme configuration information of the target scheme; Based on the scheme configuration information, a task queue to be processed for the target scheme is generated to complete the initialization of the target instance; wherein, the task queue includes multiple timed tasks to be processed; Using the target instance, the multiple pending scheduled tasks are processed to achieve the data migration of the data to be migrated; wherein, generating the pending task queue of the target scheme according to the scheme configuration information includes: In response to acquiring the database lock of the target database, determine whether the multiple pending timed tasks have been created; In response to the fact that the multiple pending timed tasks have not been created, according to the scheme configuration information, the pending timed tasks for multiple process nodes of the target scheme are created sequentially and written into the pending task queue; wherein, each process node corresponds to at least one pending timed task. The step of using the target instance to process the plurality of scheduled tasks to be processed includes: In response to the absence of a scheme control instruction, a current process node is determined from at least one process node of the target scheme; wherein the scheme control instruction is used to terminate or pause the current process node; Obtain the scheduled task corresponding to the current process node from the plurality of scheduled tasks to be processed; Determine whether the current process node in the target solution is a process node for batch submission transactions; In response to the current process node being a process node for batch submission transactions, the target instance is used to perform at least one batch processing on the data to be migrated indicated by the scheduled task to be processed corresponding to the current process node.

2. The method according to claim 1, characterized in that, One of the at least one batch processing processes includes: Determine whether a scheme control instruction for one of the batch processing methods has been received; If not, then obtain the data to be migrated from one of the batches of data to be migrated; Using the target instance, batch processing is performed on the data to be migrated in one of the batch processing processes.

3. The method according to claim 1, characterized in that, The method further includes: In response to the completion of the last batch processing in the at least one batch processing, it is determined whether the current process node is the last process node in the target scheme; In response to the current process node being the last process node, the execution status of the current process node and the execution status of the target solution are updated; In response to the fact that the current process node is not the last process node, the execution status of the current process node is updated.

4. The method according to claim 1, characterized in that, The step of using the target instance to process the plurality of scheduled tasks to be processed further includes: In response to the current process node being a process node that is not a batch-committed transaction, the processing logic of the pending timed task corresponding to the current process node is obtained. Using the target instance, and following the processing logic, the data to be migrated, as indicated by the scheduled task corresponding to the current process node, is processed.

5. The method according to claim 1, characterized in that, The target solution includes m process nodes. In response to the failure to complete the creation of the multiple pending scheduled tasks, the system sequentially creates pending scheduled tasks for multiple process nodes of the target scheme according to the scheme configuration information and writes them into the pending task queue, including: In response to the fact that the multiple pending scheduled tasks have not been created, a pending scheduled task for the current process node is created according to the scheme configuration information; Upon completion of the creation of the scheduled task to be processed for the current process node, the scheduled task to be processed for the current process node is written into the task queue to be processed. Determine whether the current process node is the last process node in the target solution; If so, then the writing process of the pending task queue ends; If not, then according to the scheme configuration information, create a scheduled task to be processed for the next process node of the current process node, so that when the scheduled task to be processed for the next process node is created, the scheduled task to be processed for the next process node is written into the scheduled task queue.

6. The method according to claim 1, characterized in that, The method further includes: In response to the fact that all the pending timed tasks have been written into the pending task queue, the database lock is released; or, Upon completion of the creation of the multiple pending scheduled tasks, the database lock is released.

7. The method according to claim 1, characterized in that, After writing the scheduled tasks to be processed into the task queue according to their execution times, the method further includes: Update the status of the target database from the initialization state to the completion state; In response to an update to the scheme configuration information, the status of the target database is updated from the completed state to the initialization state.

8. A data migration device, characterized in that, A method for implementing the data migration method as described in any one of claims 1-7 includes: The judgment module is used to determine whether the target instance running the target scheme has completed initialization in response to the target database being in the initialization state; wherein, the target scheme is used to migrate the data to be migrated in the target database; The reading module is used to read the scheme configuration information of the target scheme in response to the target instance not completing initialization; The generation module is used to generate a queue of tasks to be processed for the target scheme according to the scheme configuration information, so as to complete the initialization of the target instance; wherein, the task queue includes multiple timed tasks to be processed; The processing module is used to process the multiple scheduled tasks to be processed using the target instance, so as to realize the data migration of the data to be migrated.

Images