A data migration method and system

Abstract

The present invention discloses a data migration method and system, and relates to the field of computer technology. A specific implementation of the method includes: starting a set number of asynchronous threads, and having at least one of the asynchronous threads execute an asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object; having at least one of the asynchronous threads execute an asynchronous RESTORE command to deserialize the serialized object, and writing the obtained deserialized object into a target database; deleting the data to be migrated in the source database to complete the data migration. The method optimizes the blocking time of data migration to the source database and the target database to a constant time by executing serialization and deserialization operations through asynchronous threads, thereby avoiding the influence of the amount of data to be migrated on the blocking time.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data migration method and system. Background Art

[0002] At present, when implementing data migration of Redis database, we mainly rely on the native commands provided by Redis database, including DUMP command, RESTORE command, DEL command and MIGRATE command. Among them, MIGRATE command is the encapsulation of DUMP command, RESTORE command and DEL command. When the amount of data to be migrated is relatively large, the serialization operation in DUMP command, the deserialization operation and memory application in RESTORE command, and the memory release in DEL command are all relatively time-consuming operations. Since the native commands of Redis database are single-threaded, once a time-consuming operation occurs, data migration will be blocked, affecting normal business requests.

[0003] In the prior art, the time consumption of commands is generally saved in the following two ways: Method 1: rewrite the DUMP command and RESTORE command, modify the serialization method, and improve the efficiency of serialization and deserialization; Method 2: simulate the Redis master-slave replication process, and complete the operation of obtaining serialized data from the source database in the Forked child process, which is independent of the main process, avoiding affecting the main process's processing of normal business requests.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] (1) Method 1 will increase the size of the serialized byte stream, which will cause network congestion when the network bandwidth is relatively small. In addition, the processing speed of serialization and deserialization can only be increased by a fixed ratio. Since the original processing speed is basically proportional to the data size, as the data size increases, even the improved processing speed may still cause congestion.

[0006] (2) In the second method, when forking a child process, the database may be blocked for a short time, causing the time consumption of all requests accessing the database to increase for a short time and then quickly fall back, forming a time consumption spike. In addition, if the database receives a large number of write requests during the period from forking a child process to recycling it, the memory of the operating system will increase significantly, and a large number of memory copy operations will be generated, increasing the load on the operating system. Moreover, it can only solve the problem of source database blocking, but not the target database blocking, and is not suitable for scenarios using the MIGRATE command. Summary of the invention

[0007] In view of this, an embodiment of the present invention provides a data migration method and system, which performs serialization and deserialization operations through asynchronous threads, optimizes the blocking time of data migration on the source database and the target database to a constant time, and avoids the influence of the amount of data to be migrated on the blocking time.

[0008] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data migration method is provided.

[0009] A data migration method according to an embodiment of the present invention includes: starting a set number of asynchronous threads, at least one of the asynchronous threads executing an asynchronous DUMP command to serialize the data to be migrated of a source database to obtain a serialized object; at least one of the asynchronous threads executing an asynchronous RESTORE command to deserialize the serialized object, and writing the obtained deserialized object into a target database; and deleting the data to be migrated of the source database to complete data migration.

[0010] Optionally, the method also includes: analyzing the command implementation logic corresponding to the native commands used for data migration, and splitting the sub-logic that allows synchronous execution and the sub-logic that allows concurrent execution in the command implementation logic; assigning the sub-logic that allows synchronous execution to the main thread as a synchronous task, and assigning the sub-logic that allows concurrent execution to the asynchronous thread as an asynchronous task.

[0011] Optionally, the implementation logic of the asynchronous DUMP command includes: the main thread creates an asynchronous DUMP task and transfers the asynchronous DUMP task to the asynchronous thread; after the asynchronous thread obtains the asynchronous DUMP task, it serializes the data to be migrated and transfers the obtained serialized object to the main thread; the main thread releases the asynchronous DUMP task.

[0012] Optionally, the implementation logic of the asynchronous DUMP command includes: the main thread calculates the data volume of the data to be migrated, and determines whether the data volume of the data to be migrated is greater than or equal to a preset first threshold; if the data volume of the data to be migrated is less than the first threshold, the main thread serializes the data to be migrated; if the data volume of the data to be migrated is greater than or equal to the first threshold, the main thread creates an asynchronous DUMP task, and transfers the asynchronous DUMP task to the asynchronous thread; after the asynchronous thread obtains the asynchronous DUMP task, it serializes the data to be migrated, and transfers the obtained serialized object to the main thread; the main thread releases the asynchronous DUMP task.

[0013] Optionally, before the step of creating an asynchronous DUMP task by the main thread, the implementation logic of the asynchronous DUMP command also includes: if the amount of data to be migrated is greater than or equal to the first threshold, then according to a preset first step length, the main thread increments the reference count of the data to be migrated; before the step of releasing the asynchronous DUMP task, the implementation logic of the asynchronous DUMP command also includes: according to the first step length, the main thread decrements the reference count of the data to be migrated.

[0014] Optionally, the implementation logic of the asynchronous RESTORE command includes: the main thread creates an asynchronous RESTORE task and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transfers the obtained deserialized object to the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task.

[0015] Optionally, the implementation logic of the asynchronous RESTORE command includes: the main thread determines whether the data volume of the serialized object is greater than or equal to a preset second threshold; if the data volume of the serialized object is less than the second threshold, the main thread deserializes the serialized object and inserts the obtained deserialized object into the target database; if the data volume of the serialized object is greater than or equal to the second threshold, the main thread creates an asynchronous RESTORE task and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transfers the obtained deserialized object to the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task.

[0016] Optionally, the main thread creates an asynchronous RESTORE task, and the implementation logic of the asynchronous RESTORE command further includes: if the data volume of the serialized object is greater than or equal to the second threshold, the main thread increments the reference count of the key in the asynchronous RESTORE command and the reference count of the serialized object according to a preset second step size; before releasing the asynchronous RESTORE task, the implementation logic of the asynchronous RESTORE command further includes: according to the second step size, the main thread decrements the reference count of the key in the asynchronous RESTORE command and the reference count of the serialized object.

[0017] Optionally, inserting the deserialized object into the target database includes: determining whether an old data object exists at a corresponding position of the target database; if so, deleting the old data object and inserting the deserialized object at the position; if not, directly inserting the deserialized object at the position.

[0018] Optionally, after the step of incrementing the reference count by the main thread, the implementation logic of the asynchronous command further includes: the main thread performs write protection according to the following implementation logic: traversing all keys contained in the current write command; wherein the asynchronous command includes the asynchronous DUMP command and the asynchronous RESTORE command, and the asynchronous task includes an asynchronous DUMP task and an asynchronous RESTORE task; determining whether the data object pointed to by the current key exists and is a non-shared object, and the reference count of the data object is greater than an initial value; if the data object pointed to by the current key exists and is a non-shared object, and the reference count is greater than the initial value, terminating the traversal, returning an error prompt message to the client, and denying access; otherwise, continuing to determine the next key until the traversal is completed, and calling the native command corresponding to the current write command; after the step of decrementing the reference count by the main thread, the implementation logic of the asynchronous command further includes: canceling write protection.

[0019] Optionally, the data migration is completed by executing an asynchronous MIGRATE command; wherein the implementation logic of the asynchronous MIGRATE command includes: the main thread searches for the socket of the target database in the source database according to the IP address and port number of the target database; the main thread creates an asynchronous MIGRATE task and transfers the asynchronous MIGRATE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous MIGRATE task, it generates a byte stream of the asynchronous DUMP command and a byte stream of the asynchronous RESTORE command, sends the byte streams to the socket, blocks and waits for response data after the target database executes the asynchronous DUMP command and the RESTORE command in sequence; when the response data indicates that the asynchronous thread processing is successful, the main thread deletes the data to be migrated in the source database and releases the asynchronous MIGRATE task.

[0020] Optionally, the data migration is completed by executing an asynchronous MIGRATE command; wherein the implementation logic of the asynchronous MIGRATE command includes: the main thread calculates the data volume of the data to be migrated, and if the data volume of the data to be migrated is greater than or equal to a preset first threshold, creates an asynchronous MIGRATE task, and transmits the asynchronous MIGRATE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous MIGRATE task, it generates a byte stream of the asynchronous DUMP command and a byte stream of the asynchronous RESTORE command, sends the byte streams to the socket, blocks and waits for response data after the target database executes the asynchronous DUMP command and the RESTORE command in sequence; when the response data indicates that the asynchronous thread processing is successful, the main thread deletes the data to be migrated of the source database and releases the asynchronous MIGRATE task.

[0021] Optionally, the implementation logic of the asynchronous MIGRATE command also includes: when the byte stream fails to be sent, or the byte stream is sent successfully but the response data is not received, or the response data indicates that the asynchronous thread processing has failed, the main thread closes the socket, returns data migration failure information to the client, and releases the asynchronous MIGRATE task.

[0022] Optionally, the implementation logic of the asynchronous MIGRATE command also includes: if the amount of the data to be migrated is less than the first threshold, the main thread serializes the data to be migrated; generates a byte stream of the asynchronous RESTORE command, sends the byte stream to the socket, blocks and waits for response data after the target database executes the RESTORE command; and when the response data indicates that the asynchronous thread processing is successful, deletes the data to be migrated of the source database.

[0023] Optionally, the method further comprises: adjusting the number of current asynchronous threads according to the number of asynchronous tasks created; wherein the asynchronous tasks comprise asynchronous DUMP tasks, asynchronous RESTORE tasks and asynchronous MIGRATE tasks.

[0024] Optionally, the main thread transfers the asynchronous task to the asynchronous thread, including: acquiring a mutex lock, pushing the asynchronous task into a task queue, waking up the asynchronous thread through a conditional variable, and releasing the mutex lock; wherein the asynchronous task includes an asynchronous DUMP task, an asynchronous RESTORE task, and an asynchronous MIGRATE task; the asynchronous thread acquires the asynchronous task, including: acquiring the mutex lock, and determining whether the asynchronous task exists in the task queue; if the asynchronous task does not exist in the task queue, the asynchronous thread blocks and waits to be awakened by the main thread again; if the asynchronous task exists in the task queue, after the main thread releases the mutex lock, the asynchronous thread acquires the asynchronous task, releases the mutex lock, and determines the type of the asynchronous task to perform corresponding processing according to the type.

[0025] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data migration system is provided.

[0026] A data migration system according to an embodiment of the present invention includes: a serialization module, which is used to start a set number of asynchronous threads, and at least one of the asynchronous threads executes an asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object; a writing module, which is used to execute an asynchronous RESTORE command by at least one of the asynchronous threads to deserialize the serialized object and write the obtained deserialized object into a target database; and a deletion module, which is used to delete the data to be migrated in the source database to complete data migration.

[0027] Optionally, the system also includes: an analysis and allocation module, which is used to analyze the command implementation logic corresponding to the native commands used for data migration, and split the sub-logic allowed for synchronous execution and the sub-logic allowed for concurrent execution in the command implementation logic; and allocate the sub-logic allowed for synchronous execution as a synchronous task to the main thread, and allocate the sub-logic allowed for concurrent execution as an asynchronous task to the asynchronous thread.

[0028] Optionally, the implementation logic of the asynchronous DUMP command includes: the main thread creates an asynchronous DUMP task and transfers the asynchronous DUMP task to the asynchronous thread; after the asynchronous thread obtains the asynchronous DUMP task, it serializes the data to be migrated and transfers the obtained serialized object to the main thread; the main thread releases the asynchronous DUMP task.

[0029] Optionally, the implementation logic of the asynchronous DUMP command also includes: before the main thread creates the asynchronous DUMP task, if the amount of data to be migrated is greater than or equal to the first threshold, the main thread increases the reference count of the data to be migrated according to a preset first step length; before the main thread releases the asynchronous DUMP task, the main thread decrements the reference count of the data to be migrated according to the first step length.

[0030] Optionally, the implementation logic of the asynchronous RESTORE command includes: the main thread creates an asynchronous RESTORE task and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transfers the obtained deserialized object to the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task.

[0031] Optionally, the implementation logic of the asynchronous RESTORE command also includes: before the main thread creates the asynchronous RESTORE task, if the data volume of the serialized object is greater than or equal to the second threshold, the main thread increments the reference count of the key in the asynchronous RESTORE command and the reference count of the serialized object according to a preset second step size; before the main thread releases the asynchronous RESTORE task, the main thread decrements the reference count of the key in the asynchronous RESTORE command and the reference count of the serialized object according to the second step size.

[0032] Optionally, the system also includes: a command protection module, which is used to perform write protection by the main thread according to the following implementation logic after the main thread increments the reference count: traversing all keys contained in the current write command; wherein the asynchronous command includes the asynchronous DUMP command and the asynchronous RESTORE command, and the asynchronous task includes an asynchronous DUMP task and an asynchronous RESTORE task; determining whether the data object pointed to by the current key exists and is a non-shared object, and the reference count of the data object is greater than an initial value; if the data object pointed to by the current key exists and is a non-shared object, and the reference count is greater than the initial value, terminating the traversal, returning an error prompt message to the client, and denying access; otherwise, continuing to determine the next key until the traversal is completed, calling the native command corresponding to the current write command; and canceling the write protection after the main thread decrements the reference count.

[0033] Optionally, the system also includes: an asynchronous MIGRATE command implementation module, which is used to perform data migration according to the implementation logic of the asynchronous MIGRATE command; the implementation logic of the asynchronous MIGRATE command includes: the main thread searches for the socket of the target database in the source database according to the IP address and port number of the target database; the main thread creates an asynchronous MIGRATE task and transfers the asynchronous MIGRATE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous MIGRATE task, it generates a byte stream of the asynchronous DUMP command and a byte stream of the asynchronous RESTORE command, sends the byte stream to the socket, blocks and waits for the response data after the target database executes the asynchronous DUMP command and the RESTORE command in sequence; when the response data indicates that the asynchronous thread processing is successful, the main thread deletes the data to be migrated in the source database and releases the asynchronous MIGRATE task.

[0034] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0035] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage system for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a data migration method according to an embodiment of the present invention.

[0036] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0037] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a data migration method according to an embodiment of the present invention is implemented.

[0038] An embodiment of the above invention has the following advantages or beneficial effects: by performing serialization and deserialization operations through asynchronous threads, the blocking time of data migration on the source database and the target database is optimized to a constant time, thereby avoiding the influence of the size of the data to be migrated on the blocking time; according to whether the sub-logic allows synchronous execution, the native commands used for data migration are split, and the split results are placed in the main thread or asynchronous thread for execution, saving the time of the command; based on the value of the reference count, it is determined whether the data object is occupied by the asynchronous command. When occupied, the corresponding data is write-protected to avoid data damage; no additional memory usage is generated during the data migration process, thereby ensuring the memory availability and not affecting normal operations; the number of asynchronous threads is adjusted according to the number of asynchronous tasks to improve the efficiency of data migration.

[0039] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0041] Figure 1 is a schematic diagram of main steps of a data migration method according to an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of the main flow of a data migration method according to an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of an asynchronous module structure of a data migration method according to an embodiment of the present invention;

[0044] Figure 4 This is a logic flow chart of the asynchronous DUMP command in the third embodiment of the present invention;

[0045] Figure 5 This is a command implementation logic flow chart of the asynchronous RESTORE command in Embodiment 3 of the present invention;

[0046] Figure 6 This is a flowchart of implementing the write protection operation in Embodiment 3 of the present invention;

[0047] Figure 7 This is a logic flow chart of the command implementation of the asynchronous MIGRATE command in the third embodiment of the present invention;

[0048] Figure 8 This is a flowchart for implementing asynchronous task interaction between the main thread and the asynchronous thread in Embodiment 3 of the present invention;

[0049] Fig. 9 is a schematic diagram of main modules of a data migration system according to an embodiment of the present invention;

[0050] Fig.10 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0051] Fig.11 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0053] Figure 1 FIG. 1 is a schematic diagram of the main steps of the data migration method according to an embodiment of the present invention. Figure 1 As shown, the data migration method of the embodiment of the present invention mainly includes the following steps:

[0054] Step S101: Start a set number of asynchronous threads, and at least one of the asynchronous threads executes the asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object. Pre-analyze the command implementation logic corresponding to the native commands used for data migration, and split the sub-logic that allows synchronous execution and the sub-logic that allows concurrent execution in the command implementation logic; assign the sub-logic that allows synchronous execution to the main thread as a synchronous task, and assign the sub-logic that allows concurrent execution to the asynchronous thread as an asynchronous task. Since serialization and deserialization operations are time-consuming and run concurrently, they can be assigned to asynchronous threads. In this step, the asynchronous thread executes the serialization sub-logic in the asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object. At this time, the data to be migrated only exists in the source database.

[0055] Step S102: At least one of the asynchronous threads executes an asynchronous RESTORE command to deserialize the serialized object and write the obtained deserialized object into the target database. In the embodiment, the implementation logic of the asynchronous RESTORE command includes: the main thread creates an asynchronous RESTORE task and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transfers the obtained deserialized object to the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task. By executing the asynchronous RESTORE command according to the above implementation logic, the serialized object can be deserialized and the obtained deserialized object can be written into the target database. At this point, the data to be migrated exists in both the source database and the target database.

[0056] Step S103: Delete the data to be migrated in the source database and complete the data migration. At this point, the data to be migrated only exists in the target database, and the data migration is completed. In the above steps, both the source database and the target database can be a Redis instance. This method uses asynchronous threads to perform time-consuming operations such as serialization and deserialization, and can optimize the blocking time of data migration on the source database and the target database to a constant time, thereby avoiding the influence of the amount of data to be migrated on the blocking time.

[0057] Figure 2 FIG. 1 is a schematic diagram of the main flow of the data migration method according to an embodiment of the present invention. Figure 2 As shown, the data migration method of the embodiment of the present invention mainly includes the following steps:

[0058] Step S201: Implement an asynchronous module based on the Module function of the Redis database. The asynchronous module includes an asynchronous command module, an asynchronous thread management module and a command protection module. The asynchronous command module includes: an asynchronous DUMP command implementation module, an asynchronous RESTORE command implementation module and an asynchronous MIGRATE command implementation module. In the asynchronous command module, the command implementation logic of the DUMP command, RESTORE command and MIGRATE command used in data migration is split according to whether its sub-logic allows synchronous execution or concurrent execution. The sub-logic that allows synchronous execution is assigned to the main thread as a synchronous task, and the sub-logic that allows concurrent execution is assigned to the asynchronous thread as an asynchronous task.

[0059] In the embodiment, the command implementation logic (i.e., code logic) of the native command is analyzed. If a sub-logic involves reading and writing global variables, it belongs to the sub-logic that only allows synchronous execution, otherwise it is the sub-logic that allows concurrent execution. According to this principle, the command implementation logic of the native command can be split, and then the native command can be rewritten into the corresponding asynchronous command. The native commands here are the DUMP command, the RESTORE command, and the MIGRATE command, and the corresponding asynchronous commands are the asynchronous DUMP command, the asynchronous RESTORE command, and the asynchronous MIGRATE command.

[0060] The asynchronous thread management module is used to adjust the number of current asynchronous threads according to the number of asynchronous tasks created. When the number of asynchronous tasks is large, the number of asynchronous threads is appropriately increased; when the number of asynchronous tasks is small, some asynchronous threads are appropriately recycled to reduce the resource usage of the operating system. In the embodiment, the administrator can also determine whether the number of current asynchronous threads needs to be adjusted. If the number of concurrently executed asynchronous tasks exceeds the number of currently working asynchronous threads, the number of current asynchronous threads can be increased through the AsyncThreadSet interface. This module provides an interface for adjusting the number of current asynchronous threads. In addition, although this module cannot increase the migration speed of a single data, it improves the concurrency of multiple data migrations, and overall, it improves the data migration rate.

[0061] The command protection module is used to implement write protection operations. During the execution of asynchronous commands, if a normal write command attempts to modify the data object that the asynchronous command is accessing, the data accessed by the asynchronous command may be destroyed, causing logical abnormalities in the asynchronous command. In severe cases, the Redis service may stop running. Therefore, a write protection operation can be added to the implementation logic of the asynchronous command. When a normal write command attempts to access a data object processed by an asynchronous command, the normal write command is denied access to the data object. For the specific implementation of this module, see the subsequent section on Figure 6 Description.

[0062] Figure 3 Schematic diagram of the asynchronous module structure of the data migration method according to an embodiment of the present invention. Figure 3 As shown, the asynchronous command module includes the implementation logic of the asynchronous DUMP command, the asynchronous RESTORE command, and the asynchronous MIGRATE command. The asynchronous thread management module includes the implementation logic of the asynchronous ThreadGet command and the asynchronous ThreadSet command. The command protection module includes the implementation logic of the CommandReplace command and the CommandRecover command. The asynchronous ThreadGet command is used to set the number of asynchronous threads started. The asynchronous ThreadSet command is used to query the thread number of the current asynchronous thread to facilitate confirmation of the execution result of the asynchronous ThreadSet command. The CommandReplace command is used to add a layer of protection judgment before the native write command of Redis. The CommandRecover command is used to restore the write command of Redis to the native command.

[0063] The implementation logic of each command in the asynchronous command module is explained below.

[0064] In embodiment 1, the implementation logic of the asynchronous DUMP command may include: the main thread creates an asynchronous DUMP task and transmits the asynchronous DUMP task to the asynchronous thread; after the asynchronous thread obtains the asynchronous DUMP task, it serializes the data to be migrated and transmits the obtained serialized object to the main thread; the main thread releases the asynchronous DUMP task.

[0065] The implementation logic of the asynchronous RESTORE command may include: the main thread creates an asynchronous RESTORE task and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transfers the obtained deserialized object to the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task.

[0066] The implementation logic of the asynchronous MIGRATE command may include: the main thread searches for the socket of the target database in the source database according to the IP address and port number of the target database; the main thread creates an asynchronous MIGRATE task and transfers the asynchronous MIGRATE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous MIGRATE task, it generates a byte stream of the asynchronous DUMP command and a byte stream of the asynchronous RESTORE command, sends the byte stream to the socket, blocks and waits for the response data after the target database executes the asynchronous DUMP command and the RESTORE command in sequence; when the response data indicates that the asynchronous thread processing is successful, the main thread deletes the data to be migrated in the source database and releases the asynchronous MIGRATE task.

[0067] In the first embodiment, the amount of data to be migrated (i.e., the size of the storage space occupied) and the amount of data of the serialized object are not considered, and the serialization and deserialization operations are directly performed by the asynchronous thread. However, when the amount of data is small, data migration will not have a significant impact on the Redis service, so a corresponding threshold can be set. When the amount of data exceeds the threshold, the serialization and deserialization operations are performed by the step thread. The specific implementation is shown in the second embodiment.

[0068] In the second embodiment, the implementation logic of the asynchronous DUMP command may include: the main thread calculates the data volume of the data to be migrated, and determines whether the data volume of the data to be migrated is greater than or equal to a preset first threshold; if the data volume of the data to be migrated is less than the first threshold, the main thread serializes the data to be migrated; if the data volume of the data to be migrated is greater than or equal to the first threshold, the asynchronous DUMP command implementation logic of the first embodiment is executed.

[0069] The implementation logic of the asynchronous RESTORE command may include: the main thread determines whether the data volume of the serialized object is greater than or equal to a preset second threshold; if the data volume of the serialized object is less than the second threshold, the main thread deserializes the serialized object and inserts the obtained deserialized object into the target database; if the data volume of the serialized object is greater than or equal to the second threshold, the asynchronous RESTORE command implementation logic of Example 1 is executed.

[0070] During the execution of the asynchronous command, a layer of write protection can be added to the implementation logic of the asynchronous command, and the data in the data migration process can be write-protected according to the reference count variable to avoid data damage. See Example 3 for specific implementation.

[0071] Figure 4 FIG. 4 is a logic flow chart of the asynchronous DUMP command in the third embodiment of the present invention. Figure 4 As shown, in Example 3, the command implementation logic of the asynchronous DUMP command is as follows:

[0072] (1) The main thread determines whether the input parameters meet the requirements of the asynchronous DUMP command. If they meet the requirements, it executes (2); otherwise, it returns a prompt message indicating that the input parameters are incorrect. This step is used to determine whether the input parameters are legal.

[0073] (2) The main thread determines whether the data to be migrated specified by the input parameter exists. If so, execute (3); otherwise, return information indicating an empty object. In the embodiment, if the data to be migrated specified by the input parameter does not exist, "NOKEY" may be returned to the client.

[0074] (3) The main thread estimates the size of the storage space occupied by the data to be migrated, and determines whether the storage space occupied by the data to be migrated is greater than or equal to the set first threshold. If it is less than the first threshold, execute (4); if it is greater than or equal to the first threshold, execute (5). If the storage space occupied by the data to be migrated is less than the first threshold, it means that the data to be migrated is small, and directly executing the DUMP command will not have a significant impact on the database service. If the storage space occupied by the data to be migrated is greater than or equal to the first threshold, it means that the data to be migrated is large and needs to be processed asynchronously.

[0075] In the embodiment, the size of the storage space occupied by the data to be migrated is estimated using an application programming interface (API) provided by the Redis database.

[0076] (4) The main thread serializes the data to be migrated, returns the obtained serialized object to the client, and ends this process. In the embodiment, the serialization method provided by the Redis database is used to serialize the data to be migrated.

[0077] (5) The main thread increments the reference count of the data to be migrated according to the preset first step length to perform write protection. This step increments the reference count of the data to be migrated from the initial value by the first step length, and the first step length may be 1 in the embodiment.

[0078] (6) The main thread creates an asynchronous DUMP task and transmits the asynchronous DUMP task to the asynchronous thread through the synchronous queue.

[0079] (7) The asynchronous thread obtains the asynchronous DUMP task and serializes the data to be migrated transmitted by the asynchronous DUMP task. In the embodiment, the data to be migrated is transmitted through pointers, and a large amount of data copying will not be generated.

[0080] (8) The asynchronous thread transmits the obtained serialized object to the main thread. In the embodiment, the serialized object is transmitted through a pointer, and a large amount of data copying will not be generated.

[0081] (9) The main thread feeds the serialized object back to the client.

[0082] (10) The main thread decrements the reference count of the data to be migrated according to the first step and cancels the write protection.

[0083] (11) The main thread releases the asynchronous DUMP task, ending this process.

[0084] Figure 5 FIG. 4 is a logic flow chart of the asynchronous RESTORE command in the third embodiment of the present invention. Figure 5 As shown, in Example 3, the command implementation logic of the asynchronous RESTORE command is as follows:

[0085] (1) The main thread determines whether the input parameters meet the requirements of the asynchronous RESTORE command. If they meet the requirements, it executes (2); otherwise, it returns a prompt message indicating that the input parameters are incorrect and ends this process. This step is used to determine whether the input parameters are legal.

[0086] (2) The main thread determines whether the serialized object specified by the input parameter exists and whether replacement is allowed. If the serialized object exists and replacement is not allowed, an error message is fed back to the client and the process ends; otherwise, (3) is executed. If the serialized object exists and replacement is not allowed, an error message "BUSYKEY" can be fed back to the client.

[0087] (3) The main thread determines whether the size of the storage space occupied by the serialized object in the input parameter is greater than or equal to the set second threshold. If it is less than the second threshold, execute (4); if it is greater than or equal to the second threshold, execute (9). If the storage space occupied by the serialized object is greater than or equal to the second threshold, it indicates that asynchronous processing is required.

[0088] (4) The main thread deserializes the serialized objects in the input parameters. In the embodiment, the deserialization method provided by the Redis database is used to perform a deserialization operation on the serialized objects in the input parameters.

[0089] (5) The main thread determines whether there is an old data object at the corresponding position of the target database. If so, the old data object is deleted and the deserialized object obtained by deserialization is inserted; if not, the deserialized object is directly inserted at the position of the target database. The corresponding position refers to the position of the target database where the deserialized object is to be inserted.

[0090] (6) The main thread determines whether the input parameter contains the expiration time of the deserialized object. If it does not contain the expiration time, execute (7); if it contains the expiration time, execute (8). The expiration time is the expiration time of the deserialized object.

[0091] (7) The main thread sets the expiration time of the deserialized object and executes (8). The expiration time of the deserialized object is set when necessary according to the requirements.

[0092] (8) The main thread triggers the replication mechanism and returns a successful processing prompt to the client. According to the needs, the main thread can trigger the necessary replication mechanism, such as AOF (Append Only File), RDB (Redis DataBase), master-slave replication, etc. AOF and RDB are two persistence solutions supported by Redis.

[0093] (9) The main thread increments the reference count of the key in the input parameter and the reference count of the serialized object according to the set second step length to perform write protection. This step can prevent the release of the key and serialized object in the input parameter when the client object on the Redis server is recycled when the client actively disconnects or disconnects from the database due to network reasons.

[0094] (10) The main thread creates an asynchronous RESTORE task and transmits the asynchronous RESTORE task to the asynchronous thread through the synchronous queue.

[0095] (11) The asynchronous thread obtains the asynchronous RESTORE task and deserializes the serialized object transmitted by the asynchronous RESTORE task.

[0096] (12) The asynchronous thread transfers the obtained deserialized object to the main thread. After the deserialization is completed, the asynchronous thread transfers the obtained deserialized object to the main thread through a pointer.

[0097] (13) The main thread determines whether the old data object exists in the corresponding position of the target database. If so, it determines whether the old data object is allowed to be replaced. If so, it executes (14); if not, it executes (20); if not, it executes (15). During the deserialization process of the asynchronous thread, the main thread may create another identical data object, so it is necessary to determine again in this step whether there is a data object in the corresponding position of the target database.

[0098] (14) The main thread deletes the old data object, inserts the deserialized object into the target database, and executes (16).

[0099] (15) The main thread directly inserts the deserialized object into the corresponding position of the target database and executes (16).

[0100] (16) The main thread determines whether the input parameter contains the expiration time of the deserialized object. If it does not contain the expiration time, execute (17); if it contains the expiration time, execute (18).

[0101] (17) Set the expiration time of the deserialized object and execute (18). Set the expiration time of the deserialized object when necessary according to the requirements.

[0102] (18) The main thread triggers the replication mechanism and returns a successful processing prompt to the client. According to the needs, the main thread can trigger the necessary replication mechanism, such as AOF, RDB, master-slave replication, etc.

[0103] (19) The main thread decrements the reference count of the key in the input parameter and the reference count of the serialized object according to the second step, cancels the write protection, and executes (21).

[0104] (20) The main thread feeds back an error message to the client and executes (21). If the old data object exists and replacement is not allowed, an error message of "BUSYKEY" may be fed back to the client.

[0105] (21) The main thread releases the asynchronous RESTORE task, ending this process.

[0106] In the implementation logic of the asynchronous DUMP command and the asynchronous RESTORE command in the third embodiment, after the main thread increments the corresponding reference count, both involve write protection operations. Figure 6 The implementation process of the write protection operation is described.

[0107] Figure 6 FIG. 1 is a flowchart of implementing the write protection operation in Embodiment 3 of the present invention. Figure 6 As shown in the figure, the write protection operation in the asynchronous DUMP command and the asynchronous RESTORE command is implemented by the main thread. The specific implementation process is as follows:

[0108] (1) The main thread determines whether it has traversed all the keys contained in the current write command. If not, it executes (2); if it has, it executes (5). The write commands in this implementation logic refer to all protected write commands, including set, lpushx, lpop, restore commands, etc.

[0109] (2) The main thread determines whether the data object pointed to by the current key exists. If the data object exists and is a non-shared object, and the reference count of the data object is greater than the initial value, execute (3); otherwise, execute (4). When Redis maintains the data object pointed to by the key, in addition to the data object itself, it also maintains the type and encoding format of the data object. In the embodiment, the type and encoding format of the data object are used to determine whether it is a shared object. If the initial value of the reference count of the data object is 1, if the current reference count value is greater than 1, it means that the data object is occupied by an asynchronous command.

[0110] (3) The main thread stops traversing all the keys in the current write command, feeds back an error message to the client, denies access, and ends the process. For example, the error message "BUSYKEY" is fed back to the client.

[0111] (4) The main thread takes the next key as the current key and executes (1).

[0112] (5) The main thread transmits the input parameters from the client to the corresponding native command, which responds to the native command, thus ending the process.

[0113] Figure 7 FIG. 1 is a logic flow chart of the asynchronous MIGRATE command in the third embodiment of the present invention. Figure 7 As shown, in Example 3, the implementation logic of the asynchronous MIGRATE command is as follows:

[0114] (1) The main thread determines whether the input parameters meet the requirements of the asynchronous MIGRATE command. If they meet the requirements, it executes (2); otherwise, it returns a prompt message indicating that the input parameters are incorrect. This step is used to determine whether the input parameters are legal.

[0115] (2) The main thread determines whether the data to be migrated specified by the input parameter exists. If so, execute (3); otherwise, return information indicating an empty object. In the embodiment, if the data to be migrated specified by the input parameter does not exist, "NOKEY" may be returned to the client.

[0116] (3) The main thread searches for the socket of the target database in the source database according to the IP address and port number of the target database. A socket is a four-tuple (source IP address, source port number, target IP address, target port number), so the socket can be uniquely retrieved by the target IP address and target port number. If the socket of the target database does not exist in the source database, a socket is created and a connection is established between the source database and the target database.

[0117] (4) The main thread estimates the size of the storage space occupied by the data to be migrated, and determines whether the storage space occupied by the data to be migrated is greater than or equal to a set first threshold. If it is less than the first threshold, execute (5); if it is greater than or equal to the first threshold, execute (9).

[0118] (5) The main thread serializes the data to be migrated in the source database, generates a byte stream for the asynchronous RESTORE command, and prepares for network transmission.

[0119] (6) The main thread sends a byte stream to the found socket. If the sending fails, the current connection is closed and the socket cache in the source database is deleted, and the data migration fails.

[0120] (7) The main thread is blocked, waiting for a response from the target database. If no response is received, the current connection is closed, the socket cache in the source database is deleted, and the data migration fails. If a response is received, but the response indicates that the target database failed to execute the asynchronous RESTORE command, the data migration fails.

[0121] (8) The main thread receives a response, and the response indicates that the target database has successfully executed the asynchronous RESTORE command. Then, the data to be migrated in the source database is deleted, and a successful data migration message is returned to the client, ending the process. If there are no failures in the previous steps, it means that the data has been successfully migrated to the target database.

[0122] (9) The main thread increments the reference count of the parameter according to the set step size, creates an asynchronous MIGRATE task, and transfers the asynchronous MIGRATE task to the asynchronous thread. The parameter here can be the data to be migrated or the serialized object. The step size set for different parameters can be the same or different.

[0123] (10) The asynchronous thread obtains the asynchronous MIGRATE task, generates a byte stream of the asynchronous DUMP command corresponding to the first embodiment, and a byte stream of the asynchronous RESTORE command, and prepares for network transmission.

[0124] (11) The asynchronous thread sends a byte stream to the found socket. If the sending fails, an error mark is made.

[0125] (12) The asynchronous thread is blocked, waiting for the response from the target database. If no response is received, an error mark is made. After the target database executes the asynchronous DUMP command and RESTORE command in Example 1 in sequence, it sends the response data to the main thread. There are two types of error marks in steps (10) and (11). One type is socket error, which indicates a network error. This error requires the main thread to close the socket and recreate it the next time it is used; the other type is non-socket error, which indicates that the network is normal but a logical error occurs. The main thread only needs to return the error information returned by the target database to the client as is, without any additional processing.

[0126] (13) The main thread determines the information that needs to be responded to the client based on the error mark of the asynchronous thread. If the asynchronous thread is successful, the data to be migrated in the source database is deleted and the client is responded with a successful data migration message; otherwise, the client is responded with a failed data migration message.

[0127] (14) The main thread releases the asynchronous MIGRATE task and ends this process. If the asynchronous thread is successful, the reference count of the parameter is first decremented according to the set step size, and then the asynchronous MIGRATE task is released.

[0128] In the third embodiment, the asynchronous DUMP command, the asynchronous RESTORE command and the asynchronous MIGRATE command all involve the operation of transferring the asynchronous task from the main thread to the asynchronous thread, and the operation of the asynchronous thread obtaining the asynchronous task. Figure 8 The above process is described in detail.

[0129] Figure 8 FIG. 4 is a flowchart of implementing asynchronous task interaction between the main thread and the asynchronous thread in Embodiment 3 of the present invention. Figure 8 As shown, the main thread and the asynchronous thread interact with each other through asynchronous tasks, specifically:

[0130] (1) The main thread creates an asynchronous task. There are three types of asynchronous tasks: asynchronous DUMP tasks, asynchronous RESTORE tasks, and asynchronous MIGRATE tasks.

[0131] (2) The main thread acquires the mutex lock. The worker thread interacts with the main thread through a task queue and a set of locks. The set of locks includes a mutex lock and a condition variable. The mutex lock has two functions. The first function is to protect the task queue so that only one thread (whether the main thread or the asynchronous thread) can access it at the same time to avoid damaging the task queue. The second function is to protect the condition variable so that only one thread (whether the main thread or the asynchronous thread) can access it at the same time to avoid damaging the condition variable. The same mutex lock can only be held by one thread at a time. Other threads that attempt to acquire the mutex lock will be blocked until the mutex lock is released by the holding thread.

[0132] (3) The main thread pushes the asynchronous task created in step (1) into the task queue.

[0133] (4) The main thread wakes up the condition variable and wakes up the current asynchronous thread through the condition variable. If there are multiple asynchronous threads listening to the condition variable at this time, only one condition variable will be awakened, and which asynchronous thread is actually awakened is determined by the operating system.

[0134] (5) The main thread releases the mutex lock. The awakened asynchronous thread can then acquire the mutex lock and continue working. After the main thread releases the mutex lock, it has completed its work and can start processing the next command at any time.

[0135] (6) Creating the current asynchronous thread: In the embodiment, the current asynchronous thread is created by an asynchronous ThreadSet command.

[0136] (7) The current asynchronous thread acquires the mutex lock. This step is used to ensure the uniqueness of access rights.

[0137] (8) The current asynchronous thread determines whether there is an asynchronous task in the task queue. If there is no asynchronous task, execute (9); if there is an asynchronous task, execute (10).

[0138] (9) The current asynchronous thread is blocked, waiting for the condition variable to wake up. While waiting for the condition variable to wake up, the asynchronous thread will temporarily release the mutex lock it holds until it is successfully awakened by the condition variable, and then try to reacquire the mutex lock.

[0139] (10) The current asynchronous thread takes out the asynchronous task from the task queue. The asynchronous thread can take out the task only after the main thread releases the mutex lock. After the asynchronous task is taken out, other asynchronous threads will no longer be able to obtain the asynchronous task.

[0140] (11) The current asynchronous thread releases the mutex lock. It is a time-consuming process for an asynchronous thread to execute a specific asynchronous task, so it is best to avoid executing asynchronous tasks while holding the mutex lock.

[0141] (12) The current asynchronous thread executes the corresponding processing flow according to the type of asynchronous task. There are three types of asynchronous tasks: asynchronous DUMP task, asynchronous RESTORE task and asynchronous MIGRATE task. The corresponding processing flow can be the implementation flow of the asynchronous DUMP command, the implementation flow of the asynchronous RESTORE command and the implementation flow of the asynchronous MIGRATE command. The dotted box in the figure indicates that the subsequent processing will be carried out according to the implementation logic of the corresponding command.

[0142] This step uses the Blocked Client interface provided by the Module function of the Redis database to create an asynchronous thread and convert the serialization operation of the DUMP command, the deserialization operation of the RESTORE command, and the operations that can be executed in parallel in the MIGRATE command into asynchronous operations. Regardless of the amount of data to be migrated, the blocking of the Redis database when executing the asynchronous DUMP command, asynchronous RESTORE command, and asynchronous MIGRATE command is microseconds (<300 microseconds). When migrating a large amount of data to be migrated, the Redis service remains available.

[0143] Step S202: The client loads the asynchronous module. In the embodiment, the MODULE LOAD command provided by the Redis service is used to load the compiled asynchronous module.

[0144] Step S203: The client turns on the command protection mode and calls the COMMAND.REPLACE command (i.e., CommandReplace command) of the command protection module to add a layer of protection judgment before the native write command of Redis.

[0145] Step S204: The client starts a set number of asynchronous threads. The ASYNC.THREAD.SET command (i.e., asynchronous ThreadSet command) of the asynchronous thread management module is called to set an integer greater than 0, which represents the number of asynchronous threads to be started. In actual applications, the number of asynchronous threads is generally consistent with the number of cores of the deployment machine. For example, for an 8-core machine, 8 asynchronous threads are started to ensure that the threads are used to the maximum extent.

[0146] Step S205: The client executes the asynchronous command. Through the above steps, the asynchronous environment is ready, and the asynchronous command can be executed safely and quickly. During this period, the asynchronous commands in the asynchronous command module can be executed in any order and in any number. The asynchronous command in this step can be an asynchronous DUMP command, an asynchronous RESTORE command, and an asynchronous MIGRATE command.

[0147] Step S206: After the client confirms that the asynchronous command has been executed, it closes the asynchronous thread. When the client that executes the asynchronous command receives responses to all the asynchronous commands issued and no new asynchronous commands need to be sent, it is considered that all the asynchronous commands have been executed. In actual engineering applications, it means that all the keys that need to be migrated on the source database have been migrated, and the Redis Scan command is usually used to make this judgment. After confirming that the asynchronous command has been executed, the ASYNC.THREAD.SET command can be called again to set the number of asynchronous threads to 0 to close all asynchronous threads.

[0148] Step S207: The client turns off the command protection mode and calls the COMMAND.RECOVER command (ie, CommandRecover command) of the command protection module to restore the Redis write command to the native command.

[0149] Step S208: The client uninstalls the asynchronous module. In the embodiment, the MODULE UNLOAD command provided by the Redis service is used to uninstall the asynchronous module. In this embodiment, the data migration speed is basically the same as the migration speed of the Redis native command, and during the data migration process, the additional memory required is a very small constant value, which is independent of the amount of data to be migrated, thereby ensuring the availability of memory. In addition, data migration is usually temporary and short-term and does not need to run continuously, so the asynchronous module adopts the basic principle of "plug and play, sell after use".

[0150] It can be seen from the data migration method of the embodiment of the present invention that by executing serialization and deserialization operations through asynchronous threads, the blocking time of data migration on the source database and the target database is optimized to a constant time, thereby avoiding the influence of the amount of data to be migrated on the blocking time. At the same time, it will not increase the volume of the serialized byte stream, will not increase the load on the network bandwidth, and avoid network congestion.

[0151] Fig. 9 FIG. 1 is a schematic diagram of the main modules of the data migration system according to an embodiment of the present invention. Fig. 9 As shown, the data migration system 900 of the embodiment of the present invention mainly includes:

[0152] The serialization module 901 is used to start a set number of asynchronous threads, and at least one of the asynchronous threads executes the asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object. Pre-analyze the command implementation logic corresponding to the native commands used for data migration, and split the sub-logic that allows synchronous execution and the sub-logic that allows concurrent execution in the command implementation logic; assign the sub-logic that allows synchronous execution to the main thread as a synchronous task, and assign the sub-logic that allows concurrent execution to the asynchronous thread as an asynchronous task. Since serialization and deserialization operations are time-consuming and run concurrently, they can be assigned to asynchronous threads. In this module, the asynchronous thread executes the serialization sub-logic in the asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object. At this time, the data to be migrated only exists in the source database.

[0153] The writing module 902 is used for executing an asynchronous RESTORE command by at least one of the asynchronous threads to deserialize the serialized object and write the obtained deserialized object into the target database. In the embodiment, the implementation logic of the asynchronous RESTORE command includes: the main thread creates an asynchronous RESTORE task and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transfers the obtained deserialized object to the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task. By executing the asynchronous RESTORE command according to the above implementation logic, the serialized object can be deserialized and the obtained deserialized object can be written into the target database. At this point, the data to be migrated exists in both the source database and the target database.

[0154] The deletion module 903 is used to delete the data to be migrated in the source database and complete the data migration. In the above module, both the source database and the target database can be a Redis instance. This method uses asynchronous threads to perform time-consuming operations such as serialization and deserialization, which can optimize the blocking time of data migration on the source database and the target database to a constant time, thereby avoiding the influence of the amount of data to be migrated on the blocking time.

[0155] In addition, the data migration system 900 of the embodiment of the present invention may also include: an analysis and allocation module, a command protection module, and an asynchronous MIGRATE command implementation module. The analysis and allocation module is used to analyze the command implementation logic corresponding to the native command used for data migration, split the sub-logic that allows synchronous execution and the sub-logic that allows concurrent execution in the command implementation logic; and allocate the sub-logic that allows synchronous execution to the main thread as a synchronous task, and allocate the sub-logic that allows concurrent execution to the asynchronous thread as an asynchronous task.

[0156] A command protection module is used to perform write protection by the main thread according to the following implementation logic after the main thread increments the reference count: traverse all keys contained in the current write command; wherein the asynchronous command includes the asynchronous DUMP command and the asynchronous RESTORE command, and the asynchronous task includes the asynchronous DUMP task and the asynchronous RESTORE task; determine whether the data object pointed to by the current key exists and is a non-shared object, and the reference count of the data object is greater than an initial value; if the data object pointed to by the current key exists and is a non-shared object, and the reference count is greater than the initial value, terminate the traversal, return an error prompt message to the client, and deny access; otherwise, continue to determine the next key until the traversal is completed, and call the native command corresponding to the current write command; and cancel the write protection after the main thread decrements the reference count.

[0157] The asynchronous MIGRATE command implementation module is used to perform data migration according to the implementation logic of the asynchronous MIGRATE command. The specific implementation logic of the command is as described above.

[0158] From the above description, it can be seen that by executing serialization and deserialization operations through asynchronous threads, the blocking time of data migration on the source database and the target database is optimized to a constant time, avoiding the influence of the amount of data to be migrated on the blocking time. At the same time, it will not increase the volume of the serialized byte stream, will not increase the load on the network bandwidth, and avoid network congestion.

[0159] Fig.10 An exemplary system architecture 1000 is shown to which the data migration method or data migration system according to the embodiment of the present invention can be applied.

[0160] like Fig.10 As shown, the system architecture 1000 may include terminal devices 1001, 1002, 1003, a network 1004, and a server 1005. The network 1004 is used to provide a medium for communication links between the terminal devices 1001, 1002, 1003 and the server 1005. The network 1004 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0161] The user can use the terminal devices 1001, 1002, 1003 to interact with the server 1005 through the network 1004 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 1001, 1002, 1003.

[0162] The terminal devices 1001 , 1002 , and 1003 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0163] The server 1005 may be a server that provides various services, such as a database server that supports commands sent by users using the terminal devices 1001, 1002, and 1003. The database server may store, delete, and process data according to the received commands, and feed back the processing results (such as response information) to the terminal device.

[0164] It should be noted that the data migration method provided in the embodiment of the present application is generally executed by the terminal devices 1001 , 1002 , and 1003 , and accordingly, the data migration system is generally arranged in the terminal devices 1001 , 1002 , and 1003 .

[0165] It should be understood that Fig.10 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0166] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0167] The electronic device of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a data migration method of an embodiment of the present invention.

[0168] The computer-readable medium of the present invention stores a computer program, and when the program is executed by a processor, a data migration method according to an embodiment of the present invention is implemented.

[0169] Reference below Fig.11 , which shows a schematic diagram of the structure of a computer system 1100 suitable for implementing an electronic device of an embodiment of the present invention. Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0170] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the computer system 1100 are also stored. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0171] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0172] In particular, according to the embodiments disclosed in the present invention, the process described in the main step diagram above can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the main step diagram. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, the above-mentioned functions defined in the system of the present invention are executed.

[0173] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a serialization module, a write module, and a delete module. Among them, the names of these modules do not constitute a limitation on the module itself in certain cases, for example, the serialization module may also be described as "starting a set number of asynchronous threads, at least one of the asynchronous threads executing an asynchronous DUMP command to serialize the data to be migrated of the source database, and obtaining a module for a serialized object".

[0176] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: starting a set number of asynchronous threads, and at least one of the asynchronous threads executes an asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object; at least one of the asynchronous threads executes an asynchronous RESTORE command to deserialize the serialized object, and writes the obtained deserialized object to the target database; deleting the data to be migrated in the source database to complete the data migration.

[0177] From the above description, it can be seen that by executing serialization and deserialization operations through asynchronous threads, the blocking time of data migration on the source database and the target database is optimized to a constant time, avoiding the influence of the amount of data to be migrated on the blocking time. At the same time, it will not increase the volume of the serialized byte stream, will not increase the load on the network bandwidth, and avoid network congestion.

[0178] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0179] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data migration method, characterized in that: include: A set number of asynchronous threads are started, and at least one of the asynchronous threads executes an asynchronous DUMP command to serialize the data to be migrated of the source database to obtain a serialized object; wherein the implementation logic of the asynchronous DUMP command includes: the main thread calculates the data volume of the data to be migrated, and determines whether the data volume of the data to be migrated is greater than or equal to a preset first threshold; if the data volume of the data to be migrated is greater than or equal to the first threshold, the main thread creates an asynchronous DUMP task and transmits the asynchronous DUMP task to the asynchronous thread; after the asynchronous thread obtains the asynchronous DUMP task, it serializes the data to be migrated and transmits the obtained serialized object to the main thread; the main thread releases the asynchronous DUMP task; before the step of the main thread creating the asynchronous DUMP task, it also includes: according to the preset first step length, the main thread increments the reference count of the data to be migrated; before the step of releasing the asynchronous DUMP task, it also includes: according to the first step length, the main thread decrements the reference count of the data to be migrated; At least one of the asynchronous threads executes an asynchronous RESTORE command to deserialize the serialized object, and writes the obtained deserialized object into a target database; Delete the data to be migrated in the source database to complete the data migration.

2. The method according to claim 1, characterized in that The method further comprises: Analyze the command implementation logic corresponding to the native commands used for data migration, and separate the sub-logic that allows synchronous execution and the sub-logic that allows concurrent execution in the command implementation logic; The sub-logic that allows synchronous execution is assigned to the main thread as a synchronous task, and the sub-logic that allows concurrent execution is assigned to the asynchronous thread as an asynchronous task.

3. The method according to claim 2, characterized in that The implementation logic of the asynchronous DUMP command includes: The main thread creates an asynchronous DUMP task and transmits the asynchronous DUMP task to the asynchronous thread; After the asynchronous thread obtains the asynchronous DUMP task, the asynchronous thread serializes the data to be migrated, and transmits the obtained serialized object to the main thread; The main thread releases the asynchronous DUMP task.

4. The method according to claim 1, characterized in that: The implementation logic of the asynchronous DUMP command includes: If the amount of the data to be migrated is less than the first threshold, the main thread serializes the data to be migrated.

5. The method according to claim 2, characterized in that: The implementation logic of the asynchronous RESTORE command includes: The main thread creates an asynchronous RESTORE task and transmits the asynchronous RESTORE task to the asynchronous thread; After the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transmits the obtained deserialized object to the main thread; The main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task.

6. The method according to claim 2, characterized in that The implementation logic of the asynchronous RESTORE command includes: The main thread determines whether the data volume of the serialized object is greater than or equal to a preset second threshold; If the data volume of the serialized object is less than the second threshold, the main thread deserializes the serialized object and inserts the obtained deserialized object into the target database; If the data volume of the serialized object is greater than or equal to the second threshold, the main thread creates an asynchronous RESTORE task and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, it deserializes the serialized object and transfers the obtained deserialized object to the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task.

7. The method according to claim 6, characterized in that The main thread creates an asynchronous RESTORE task, and the implementation logic of the asynchronous RESTORE command also includes: If the data volume of the serialized object is greater than or equal to the second threshold, the main thread increments the reference count of the key in the asynchronous RESTORE command and the reference count of the serialized object according to a preset second step size; Before releasing the asynchronous RESTORE task, the implementation logic of the asynchronous RESTORE command further includes: According to the second step length, the main thread decrements the reference count of the key in the asynchronous RESTORE command and the reference count of the serialized object.

8. The method according to claim 1 or 7, characterized in that: After the step of incrementing the reference count by the main thread, the implementation logic of the asynchronous command further includes: the main thread performs write protection according to the following implementation logic: Traverse all keys contained in the current write command; wherein the asynchronous command includes the asynchronous DUMP command and the asynchronous RESTORE command, and the asynchronous task includes the asynchronous DUMP task and the asynchronous RESTORE task; Determine whether the data object pointed to by the current key exists and is a non-shared object, and the reference count of the data object is greater than the initial value; If the data object pointed to by the current key exists and is a non-shared object, and the reference count is greater than the initial value, the traversal is terminated, an error message is returned to the client, and access is denied; otherwise, the next key is determined until the traversal is completed, and the native command corresponding to the current write command is called; After the step of the main thread decrementing the reference count, the implementation logic of the asynchronous command further includes: canceling the write protection.

9. The method according to claim 2, characterized in that: The data migration is completed by executing an asynchronous MIGRATE command; wherein the implementation logic of the asynchronous MIGRATE command includes: The main thread searches the source database for the socket of the target database according to the IP address and port number of the target database; The main thread creates an asynchronous MIGRATE task and transfers the asynchronous MIGRATE task to the asynchronous thread; After acquiring the asynchronous MIGRATE task, the asynchronous thread generates a byte stream of the asynchronous DUMP command and a byte stream of the asynchronous RESTORE command, sends the byte stream to the socket, blocks and waits for response data after the target database executes the asynchronous DUMP command and the RESTORE command in sequence; When the response data indicates that the asynchronous thread processing is successful, the main thread deletes the data to be migrated in the source database and releases the asynchronous MIGRATE task.

10. The method according to claim 3, characterized in that: The data migration is completed by executing an asynchronous MIGRATE command; wherein the implementation logic of the asynchronous MIGRATE command includes: The main thread calculates the data volume of the data to be migrated, and if the data volume of the data to be migrated is greater than or equal to a preset first threshold, creates an asynchronous MIGRATE task, and transmits the asynchronous MIGRATE task to the asynchronous thread; After acquiring the asynchronous MIGRATE task, the asynchronous thread generates a byte stream of the asynchronous DUMP command and a byte stream of the asynchronous RESTORE command, sends the byte stream to the socket, blocks and waits for response data after the target database executes the asynchronous DUMP command and the RESTORE command in sequence; When the response data indicates that the asynchronous thread processing is successful, the main thread deletes the data to be migrated in the source database and releases the asynchronous MIGRATE task.

11. The method according to claim 9 or 10, characterized in that: The implementation logic of the asynchronous MIGRATE command also includes: When the byte stream fails to be sent, or the byte stream is sent successfully but the response data is not received, or the response data indicates that the asynchronous thread processing has failed, the main thread closes the socket, returns data migration failure information to the client, and releases the asynchronous MIGRATE task.

12. The method according to claim 10, characterized in that The implementation logic of the asynchronous MIGRATE command also includes: If the amount of the data to be migrated is less than the first threshold, the main thread serializes the data to be migrated; Generate a byte stream of the asynchronous RESTORE command, send the byte stream to the socket, block and wait for response data after the target database executes the RESTORE command; When the response data indicates that the asynchronous thread processing is successful, the data to be migrated in the source database is deleted.

13. The method according to any one of claims 3 to 7 or 9 to 10, characterized in that: The method further includes: adjusting the number of current asynchronous threads according to the number of created asynchronous tasks; wherein the asynchronous tasks include asynchronous DUMP tasks, asynchronous RESTORE tasks and asynchronous MIGRATE tasks.

14. The method according to any one of claims 3 to 7 or 9 to 10, characterized in that: The main thread transmits the asynchronous task to the asynchronous thread, including: Acquire a mutex lock, push the asynchronous task into a task queue, wake up the asynchronous thread through a condition variable, and release the mutex lock; wherein the asynchronous task includes an asynchronous DUMP task, an asynchronous RESTORE task, and an asynchronous MIGRATE task; The asynchronous thread acquires the asynchronous task including: Acquire the mutex lock, and determine whether the asynchronous task exists in the task queue; If the asynchronous task does not exist in the task queue, the asynchronous thread is blocked and waits to be awakened by the main thread again; If the asynchronous task exists in the task queue, after the main thread releases the mutex lock, the asynchronous thread acquires the asynchronous task, releases the mutex lock, determines the type of the asynchronous task, and performs corresponding processing according to the type.

15. A data migration system, characterized in that: include: A serialization module is used to start a set number of asynchronous threads, and at least one of the asynchronous threads executes an asynchronous DUMP command to serialize the data to be migrated in the source database to obtain a serialized object; A writing module, configured to execute an asynchronous RESTORE command by at least one of the asynchronous threads to deserialize the serialized object and write the obtained deserialized object into a target database; The deletion module is used to delete the data to be migrated in the source database to complete the data migration.

16. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 14.

17. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images