Unlock instant, AI-driven research and patent intelligence for your innovation.

Data migration method and system

A technology for data migration, which is applied in the computer field and can solve problems such as time-consuming spikes, database blocking, and source database blocking

Pending Publication Date: 2021-01-19
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] (1) Method 1 will increase the size of the serialized byte stream, which will lead to network congestion when the network bandwidth is relatively small; in addition, the processing speed of serialization and deserialization can only be increased by a fixed ratio , since the original processing speed is basically proportional to the data size, as the data size increases, even the improved processing speed may still cause congestion
[0006] (2) In method 2, when the Fork child process is in progress, the database may be blocked for a short time, causing all requests to access the database to rise for a short time and then fall back quickly, forming a time-consuming spike; After the process is recycled, if the database receives more write requests, the memory of the operating system will increase significantly, and a large number of memory copy operations will be generated, which will increase the load of the operating system; and it can only solve the source database blocking The problem cannot solve the blocking of the target database, and it is not applicable to the scenario of using the MIGRATE command

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data migration method and system
  • Data migration method and system
  • Data migration method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0064] In Embodiment 1, the implementation logic of the asynchronous DUMP command may include: the main thread creates an asynchronous DUMP task, and transmits the asynchronous DUMP task to the asynchronous thread; after the asynchronous thread obtains the asynchronous DUMP task, serializes the data to be migrated, and the The object is transferred to the main thread; the main thread releases the asynchronous DUMP task.

[0065] The implementation logic of the asynchronous RESTORE command may include: the main thread creates an asynchronous RESTORE task, and transfers the asynchronous RESTORE task to the asynchronous thread; after the asynchronous thread obtains the asynchronous RESTORE task, deserializes the serialized object, and transfers the obtained deserialized object To the main thread; the main thread inserts the deserialized object into the target database and releases the asynchronous RESTORE task.

[0066] The implementation logic of the asynchronous MIGRATE command...

Embodiment 2

[0068] In Embodiment 2, the implementation logic of the asynchronous DUMP command may include: the main thread calculates the data volume of the data to be migrated, and judges whether the data volume of the data to be migrated is greater than or equal to a preset first threshold; if the data volume of the data to be migrated is less than the first threshold, the main thread serializes the data to be migrated; if the data volume of the data to be migrated is greater than or equal to the first threshold, the asynchronous DUMP command in Embodiment 1 is executed to implement the logic.

[0069] The implementation logic of the asynchronous RESTORE command may include: the main thread judges whether the data volume of the serialized object is greater than or equal to the preset second threshold; if the data volume of the serialized object is less than the second threshold, the main thread deserializes the serialized object and insert the obtained deserialized object into the target...

Embodiment 3

[0106] In the third embodiment, in the implementation logic of the asynchronous DUMP command and the asynchronous RESTORE command, after the main thread increments the corresponding reference count, both involve write protection operations, which are combined below Image 6 Describe the implementation process of the write protection operation.

[0107] Image 6 It is a flow chart of realizing the write protection operation in the third embodiment of the present invention. Such as Image 6 As shown, the write protection operation in the asynchronous DUMP command and the asynchronous RESTORE command is implemented by the main thread, and the specific implementation process is as follows:

[0108] (1) The main thread judges whether all the keys contained in the current write command have been traversed, and if the traverse has not been completed, execute (2); if the traverse has been completed, execute (5). The write commands in this implementation logic refer to all protected...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data migration method and system, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of starting a set number of asynchronous threads, and executing an asynchronous DUMP command by at least one asynchronous thread to serialize to-be-migrated data of a source database to obtain a serialized object; executing an asynchronous RESTORE command by at least one asynchronous thread so as to deserialize the serialized object, and writing the obtained deserialized object into a target database; and deleting the to-be-migrated data of the source database to complete data migration. Serialization and deserialization operations are executed through an asynchronous thread, the blocking time of data migration to a source database and a target database is optimized to constant time, and the influence of the data size of to-be-migrated data on the blocking time is avoided.

Description

technical field [0001] The invention relates to the field of computers, in particular to a data migration method and system. Background technique [0002] At present, when implementing the data migration of the Redis database, it mainly relies on the native commands provided by the Redis database, including the DUMP command, the RESTORE command, the DEL command and the MIGRATE command. Among them, the MIGRATE command is the encapsulation of the DUMP command, the RESTORE command and the DEL command. When the amount of migrated data is relatively large, the serialization operation in the DUMP command, the deserialization operation and memory application in the RESTORE command, and the memory release in the DEL command are all time-consuming operations. Since the native commands of the Redis database run on a single thread, once time-consuming operations occur, data migration will be blocked and normal business requests will be affected. [0003] In the prior art, the followi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/21G06F9/48
CPCG06F9/4881G06F2209/486G06F16/214
Inventor 黄东宏聂启忠
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD