Data migration method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data migration method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring each database table of a source end database, and querying a target slice size of the database table; the target slice size is determined according to the average slice size of the database table, the data migration configuration parameters and the available memory of the server; according to the target slice size, slicing each database table to obtain slice information; and obtaining target migration data according to the slice information, and writing the target migration data into the target end database. By adopting the method, the performance problems of insufficient storage space or abnormal exit and the like can be avoided, so that the robustness of a data migration program is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of big data technology, the demand for database stability and availability is increasing day by day. Data migration technology has emerged to meet scenarios such as cloud migration, backup and disaster recovery, capacity expansion and contraction, archiving, and data analysis.

[0003] In traditional technology, data migration is performed through traditional export and import tools such as mongoexport, mongoimport, mongodump, and mongostore.

[0004] However, since the performance parameters of the server are fixed, in the case of large-scale data volumes, performance issues such as insufficient storage space or abnormal exit due to limited file system performance may occur. Summary of the Invention

[0005] Based on this, it is necessary to provide a data migration method, device, computer equipment, computer-readable storage medium and computer program product that can avoid performance problems such as insufficient storage space or abnormal exit and improve the robustness of the data migration program in response to the above technical problems.

[0006] In a first aspect, the present application provides a data migration method, the method comprising:

[0007] Obtain each database table of the source database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and available memory of the server;

[0008] Slicing each of the database tables according to the target slice size to obtain slice information;

[0009] Target migration data is obtained according to the slice information, and the target migration data is written into a target-side database.

[0010] In one embodiment, the data migration configuration parameters include the number of full threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the number of copies on the source end and the number of copies on the target end; and the method for determining the target slice size includes:

[0011] Determining a total number of copies based on the number of copies on the source end and the number of copies on the target end;

[0012] Determining a maximum slice size based on available memory of the server, the number of full threads, the data expansion factor, and the total number of copies;

[0013] A target slice size is determined according to the average slice size and the maximum slice size.

[0014] In one embodiment, obtaining target migration data according to the slice information and writing the target migration data into a target-side database includes:

[0015] Obtaining slice serialization data according to the slice information, and storing the slice serialization data in intermediate storage;

[0016] According to the target slice size, obtaining serialized data of each slice from the intermediate storage;

[0017] Target migration data is obtained based on the slice serialization data, and the target migration data is written into the target end database.

[0018] In one embodiment, the database table carries data to be migrated; obtaining slice serialization data according to the slice information, and storing the slice serialization data in an intermediate storage includes:

[0019] Reading corresponding data to be migrated from the database according to the slice information;

[0020] Serializing the data to be migrated to obtain slice serialized data;

[0021] Storing each of the slice serialized data in the intermediate storage in sequence.

[0022] In one embodiment, the slice information includes range information of a database table; and reading corresponding data to be migrated from the database according to the slice information includes:

[0023] Reading corresponding data to be migrated from each database table according to range information of the corresponding database table by each of the source threads in the plurality of source threads;

[0024] The serializing the data to be migrated to obtain slice serialized data includes:

[0025] Performing a serialization operation on the data to be migrated by each source-end thread to obtain slice serialization data corresponding to each source-end thread;

[0026] The intermediate storage includes a plurality of queues; the queues correspond one-to-one to the source threads; and storing each slice serialized data in the intermediate storage in sequence includes:

[0027] According to the order of each database table, each source-end thread stores the corresponding slice serialized data into the queue corresponding to the source-end thread.

[0028] In one embodiment, obtaining the serialized data of each slice from the intermediate storage according to the target slice size includes:

[0029] Obtaining each slice serialized data from a target queue through each target thread among a plurality of target threads; wherein the source thread corresponds to the target thread one-to-one, and the target queue represents a queue corresponding to the source thread and the target thread;

[0030] Obtaining target migration data according to the slice serialization data, and writing the target migration data into the target end database includes:

[0031] Each target-side thread performs a deserialization operation on each slice serialized data to obtain the target migration data corresponding to each target-side thread, and each target-side thread writes the target migration data into the target-side database.

[0032] In a second aspect, the present application further provides a data migration device, the device comprising:

[0033] A source-side reading module is used to obtain each database table of the source-side database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and available memory of the server;

[0034] It is also used to slice each of the database tables according to the target slice size to obtain slice information;

[0035] Also used to obtain target migration data according to the slice information;

[0036] The target-side writing module is used to write the target migration data into the target-side database.

[0037] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0039] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0040] The above-described data migration method, apparatus, computer device, computer-readable storage medium, and computer program product determine a target slice size before data migration based on the average slice size of the database table, data migration configuration parameters, and the available memory of the server. Specifically, the target slice size for data migration is determined based on server performance parameters. By querying the target slice size for the database table, each database table is segmented according to the target slice size. Target migration data is obtained based on the slice information and written to the target database. Server performance parameters are taken into account when slicing each database table, thereby avoiding abnormal interruptions during the data migration process due to insufficient server performance parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is an application environment diagram of a data migration method in one embodiment;

[0043] Figure 2 1 is a flow chart of a data migration method according to an embodiment;

[0044] Figure 3 A schematic flow chart of a method for determining a target slice size according to an embodiment;

[0045] Figure 4 A schematic diagram of a process for writing target migration data into a target-side database in one embodiment;

[0046] Figure 5 A schematic diagram of a process for storing slice serialized data into intermediate storage in one embodiment;

[0047] Figure 6 A schematic diagram of modules included in a server for executing a data migration method in one embodiment;

[0048] Figure 7 This is a flowchart of a method for performing full data migration using multiple threads on the source side in one embodiment;

[0049] Figure 8A flowchart of a method for performing full data migration using multiple threads on the source side and multiple threads on the target side in one embodiment is shown;

[0050] Figure 9 A flowchart of a method for performing full data migration using multiple threads on a target side according to one embodiment is shown;

[0051] Figure 10 This is a schematic diagram of the system architecture of an embodiment;

[0052] Figure 11 is a structural block diagram of a data migration device in one embodiment;

[0053] Figure 12 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] The data migration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the source database 106 communicates with the server 102 through the network. The source database 106 stores the data to be migrated. The source database 106 can be integrated on the server 102, or it can be placed on the cloud or other network servers. The source database 106 can be a source database, or it can be multiple source databases. The target database 104 communicates with the server 102 through the network. The target database 104 stores the migrated data. The target database 104 can be integrated on the server 102, or it can be placed on the cloud or other network servers. The server 102 obtains each database table of the source database and queries the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, the data migration configuration parameters and the available memory of the server; according to the target slice size, each database table is sliced ​​to obtain slice information; according to the slice information, the target migration data is obtained, and the target migration data is written into the target database. Among them, the server 102 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud computing services.

[0056] In an exemplary embodiment, Figure 2 As shown, a data migration method is provided, which is applied to Figure 1 The server in the example is used to illustrate the method, which includes the following steps S202 to S206.

[0057] Step S202 , obtaining each database table of the source database, and querying the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and available memory of the server.

[0058] Data migration refers to copying the structural information and data of objects in a database from one database system to another. A full migration is the process of migrating all data from a source database at a specific point in time to a target database, including all table structures, records, indexes, and other content. The source database refers to the data sending end during data migration. The target database refers to the data receiving end during data migration. The target slice size is a measure of the granularity of a database table. The target slice size is measured in MB to prevent out-of-memory (OOM) errors caused by a single slice being too large, which can exceed the memory limit.

[0059] Optionally, for example, in a full migration scenario, before obtaining the database tables of the source database, that is, before migrating the data, a structural migration is performed. Structural migration refers to migrating the structural information of database objects. For example, objects such as tables, views, functions, and stored procedures. The server obtains the server's account and connection information for the source and target databases through the configuration module, such as username, password, IP address, port, and connection string. Based on the account and connection information, the server establishes a communication connection with the source and target databases. The server obtains the structural information of the source database; generates corresponding Data Definition Language (DDL) statements based on the database's structural information; executes each DDL statement in the target database for each DDL statement, and writes the structural information of the source database to the target database. If the database is a Mongo database, after establishing a connection to the source database through the MongoDB driver, information including indexes and table creation statements is obtained through MongoDB's built-in commands such as listIndexes. The server replays the structural information of the source database in the target database to complete the structural migration.

[0060] Optionally, after completing the structure migration, data migration is also required. The server obtains each database table of the source database, including: obtaining the data to be migrated from the source database; and obtaining each database table matching the data to be migrated.

[0061] In an optional embodiment, the server determines, based on the hash value of the data to be migrated, that each source thread obtains a database table that matches the data to be migrated. For example, if there are 100 database tables in the source database that match the data to be migrated, the server obtains all 100 database tables.

[0062] In an optional embodiment, each of the server's multiple source threads determines, based on a hash value of the data to be migrated, that each source thread acquires a database table that matches the data to be migrated. At any given moment, one of the source threads executes the task of acquiring the database table. For example, if there are 100 database tables in the source database that match the data to be migrated, and the server has 10 threads, thread 1 acquires tables 1, 3, 6, ..., 100; thread 2 acquires tables 2, 5, 7, ..., 99; ... thread 10 acquires tables xx, ..., xx. When thread 1 acquires database table 1 from the source database, thread 1 is locked, preventing other threads from acquiring the corresponding database table from the source database, thus preventing multiple threads from acquiring the same database table simultaneously.

[0063] Optionally, the server obtains each database table of the source database and queries the target slice size of the database table. The target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and available memory of the server.

[0064] Step S204: Slice each database table according to the target slice size to obtain slice information.

[0065] A slice is a database table with defined start and end points, allowing you to read a specific section of data from the table. Granularity controls the size of the slice.

[0066] Optionally, the server slices each database table according to the target slice size to obtain slice information, wherein the slice information is the start point and end point information of the database table, that is, the slice information is a kind of range information.

[0067] Optionally, when there are multiple source threads, each source thread pool includes a slicing thread. For each slicing thread corresponding to a source read thread, each database table is sliced ​​according to the target slice size to obtain slice information for each source thread. The number of source threads is configured via the configuration module. It is understood that performance parameters such as memory usage, number of concurrent threads, and slice size must all be pre-configured in the configuration file.

[0068] Step S206: Obtain target migration data according to the slice information, and write the target migration data into the target-end database.

[0069] Optionally, the server processes the slice information to obtain target migration data and writes the target migration data to the target database. For example, the server obtains the corresponding data to be migrated based on the slice information, serializes and deserializes the data to be migrated, and obtains the target migration data. The server then writes the target migration data in batches to the target database.

[0070] In the above data migration method, before data migration begins, the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and the server's available memory. In other words, the target slice size for data migration is determined based on server performance parameters. By querying the target slice size for the database table, each database table is segmented according to the target slice size. The target migration data is then obtained based on the slice information and written to the target database. By taking server performance parameters into account when slicing each database table, unexpected interruptions during the data migration process due to insufficient server performance parameters are avoided.

[0071] In an exemplary embodiment, Figure 3 As shown, the data migration configuration parameters include the number of full threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the number of copies at the source end and the number of copies at the target end; the method for determining the target slice size includes steps S302 to S306.

[0072] Step S302: Determine the total number of copies based on the number of copies at the source end and the number of copies at the target end.

[0073] The full number of threads refers to the maximum number of threads that the server can run. The number of source-side copies refers to the number of copies made from the slice serialized data read from the source database to the storage process. For example, database queries, serialization, data processing, and other operations all generate copies. The number of target-side copies refers to the number of copies made from the slice serialized data read from storage to the target-side database, such as data processing operations such as deserialization. Typically, the number of copies on the source or target side is 3. The data expansion coefficient refers to the fact that the document data stored in the database, such as MongoDB, may have a multi-layer nested structure. As a result, after data processing, the size of the serialized instance object may be several times larger than the text data itself, and is usually set between 5-10 times.

[0074] Optionally, in an architecture where the source database is connected to an intermediate storage; and the intermediate storage is connected to a target database, the total number of copies is equal to the number of source copies, the number of target copies, and the number of copies of one intermediate storage.

[0075] Step S304: Determine the maximum slice size based on the available memory of the server, the number of full threads, the data expansion coefficient, and the total number of copies.

[0076] Optionally, after the server obtains specific values ​​of its own available memory, configured full thread number, data expansion coefficient, and total number of copies, it can substitute them into the first relational expression, wherein the first relational expression is determined based on the second relation.

[0077] The second relational equation is: The maximum total memory occupied by the program at a given moment is approximately equal to the number of full threads on the source side x {maximum chunk size maxChunkSize x expansion factor x (number of copies on the source side + 1)} + the number of full threads on the target side x (maximum chunk size maxChunkSize x expansion factor x number of copies on the target side). In an architecture where the source database is connected to intermediate storage, and the intermediate storage is connected to the target database, the number of full threads on the source side = the number of full threads on the target side = the number of full threads. Translating the second relational equation, the maximum total memory occupied by the program at a given moment is approximately equal to the number of full threads × maximum chunk size maxChunkSize × expansion factor × (number of copies on the source side + 1 + number of copies on the target side). Based on the second relational equation, the server determines the first relation: maximum memory / total number of copies / expansion factor / number of full threads is approximately equal to the maximum chunk size maxChunkSize. For example, taking the smallest 1C4G machine with an expansion factor of 5, the number of copies on the source and target sides is 3, and the number of full threads is 1. If the configuration is set to run at 90% of the memory, then the maximum MaxChunkSize = 4G x 1024x0.9 / (3+1+3) / 5 = 105MB.

[0078] Step S306: Determine the target slice size according to the average slice size and the maximum slice size.

[0079] Among them, the average slice size AVchunkSize in a certain database is 64MB.

[0080] Alternatively, the server typically splits table data into multiples of AVchunkSize for higher efficiency, so the target chunk size is typically chosen to be an integer multiple of (AVchunkSize, maxChunkSize). If the server has limited available memory, you can specify an average chunk size, such as 64MB.

[0081] In this embodiment, the target slice size can be determined by the data migration configuration parameters and the average slice size. On the one hand, dividing each database table according to the target slice size can avoid data migration interruption caused by insufficient performance. On the other hand, dividing each database table according to the target slice size is more efficient.

[0082] In an exemplary embodiment, Figure 4As shown, the target migration data is obtained according to the slice information, and the target migration data is written into the target end database, including steps S402 to S406.

[0083] Step S402: Obtain slice serialization data according to the slice information, and store the slice serialization data in an intermediate storage.

[0084] Optionally, the server reads the data to be migrated according to the slice information, and serializes the data to be migrated to obtain slice serialized data, and stores the slice serialized data in an intermediate storage in a first-in-first-out manner.

[0085] Step S404: Obtain serialized data of each slice from the intermediate storage according to the target slice size.

[0086] Optionally, the server pulls multiple batches of slice serialization data from the tail of the intermediate storage queue according to the target slice size.

[0087] Step S406: Obtain target migration data based on the slice serialization data, and write the target migration data into the target-end database.

[0088] Optionally, the server deserializes the pulled slice serialized data. For example, if the database is MongoDB, the server converts the field description metadata and binary stream value back to the BsonDocument type recognizable by MongoDB, thereby obtaining the target migration data.

[0089] In this embodiment, slice serialization data is obtained based on the slice information, and the slice serialization data is first stored in the intermediate storage for subsequent target-side writing, thereby decoupling the source-side reading and serialization from the target-side writing, further improving the controllability of performance.

[0090] In an exemplary embodiment, Figure 5 As shown, the database table carries the data to be migrated; according to the slice information, the slice serialization data is obtained, and the slice serialization data is stored in the intermediate storage, including steps S502 to S506.

[0091] Step S502: Read the corresponding data to be migrated from the database table according to the slice information.

[0092] Before reading the corresponding data to be migrated from the database according to the slice information, the server includes a configuration module, a scheduling module, a structure migration module, a source end reading module, an intermediate storage module and a target reading and writing module, such as Figure 6As shown. Among them, the configuration module is used for source-side information configuration, target-side information configuration, migration object configuration, intermediate storage configuration and performance parameter configuration. The server loads the settings required by other modules from the configuration file through the configuration module, such as the account and connection information of the source database and the target database of the migration module, such as user name, password, IP address port, connection string, etc. Parameter indicators related to the performance of the migration module, such as memory occupied, number of concurrent threads, number of batches, and slice size. The intermediate storage configuration can specify the use of memory, local file system, or implement interface expansion to other storage forms. The migration object is a streamlined data model that the server migrates from the source database according to instructions. It is specified to the specific library table column in json format with wildcards. At the same time, filtering rules, name mapping, etc. can also be set.

[0093] The scheduling module is responsible for loading configurations, sequentially scheduling structure migrations, starting source-side read modules and target-side write modules, and managing the lifecycle of each module. It terminates the program when all source-side read modules and target-side write modules have been completed. It is also responsible for writing progress information and determining the logic for recovery from abnormal interruptions.

[0094] The structure migration module retrieves source database information from the configuration module. After establishing a database connection using the MongoDB driver, it uses built-in MongoDB commands such as listIndexes to retrieve information such as indexes and table creation statements. For MongoDB shard sets, it also queries the config repository for shard keys matching the migration target. After reading the source database structure information, it records it as DDL statements and replays them in the target database.

[0095] The source-side reading module is used to obtain source-side database information and migration objects from the configuration module. For a replica set, after establishing a database connection through the MongoDB driver, it queries the database for information such as libraries and tables that match the migration objects, performing segmented queries at the granularity of the table (collection) to be migrated.

[0096] The intermediate storage module is used to store shard serialized data and decouple the source-side reading module from the target-side writing module.

[0097] The target-side write module is used to obtain concurrent configuration parameters from the configuration module, and then use multiple threads to concurrently pull serialized batches of sharded data from the tail of the bidirectional queue of the intermediate storage module.

[0098] Optionally, the server reads corresponding data to be migrated from each database according to the slice information.

[0099] Step S504: serialize the data to be migrated to obtain slice serialized data.

[0100] Optionally, the server serializes the migrated data to make it compatible with different database and table structures. The serialized intermediate data model includes field descriptions and actual data. The field description mainly contains metadata information of fields such as the primary key (_id), unique key, and binary stream type. Correspondingly, the actual data needs to store the specific value of the primary key, unique key, and text converted into a binary stream corresponding to a database record.

[0101] Taking the database MangoDB as an example, the server's text-to-binary algorithm uses BsonDocumentCodec to convert BsonDocument into a binary stream array byte[], which is then encapsulated using ByteBuffer to obtain sliced ​​serialized data.

[0102] Step S506: Store the serialized data of each slice in the intermediate storage in sequence.

[0103] Optionally, serialize each slice into an intermediate storage in order. The intermediate storage is based on a concurrent data structure called a thread-safe ConcurrentMap and a bidirectional blocking queue to store data in an intermediate format. The source side inserts data into the queue head, and the destination side writes data from the queue tail.

[0104] In this embodiment, data is serialized to be compatible with different library table structures, and each slice serialized data is stored in an intermediate storage to achieve decoupling from subsequent writing programs.

[0105] In an exemplary embodiment, a method for performing full data migration by multi-threading on the source side is as follows: Figure 7 As shown, it includes steps S702 to S706, the slicing information includes the range information of the database table; reading the corresponding data to be migrated from the database according to the slicing information, including step S702, through each source thread in multiple source threads, reading the corresponding data to be migrated from each database table according to the range information of the corresponding database table.

[0106] Optionally, the server reads the corresponding data to be migrated from each database table through each source thread in the multiple source threads according to the range information of the corresponding database table. Figure 8 As shown, the server uses n source-side threads to concurrently read the corresponding data to be migrated from each database table according to the range information of the database table.

[0107] The data to be migrated is serialized to obtain slice serialized data, including step S704 , in which the data to be migrated is serialized by each source-end thread to obtain slice serialized data corresponding to each source-end thread.

[0108] Optionally, multiple source threads perform serialization operations on the data to be migrated to obtain slice serialization data corresponding to each source thread. Figure 8 As shown, n source threads of the server perform serialization operations on the data to be migrated to obtain slice serialization data corresponding to each source thread.

[0109] The intermediate storage includes multiple queues; the queues correspond one-to-one to the source threads; each slice serialized data is stored in the intermediate storage in sequence, including step S706: in the order of each database table, each source thread stores the corresponding slice serialized data in the queue corresponding to the source thread through each source thread.

[0110] Optionally, the server stores the corresponding slice serialized data in the queue corresponding to the source thread through each source thread. Figure 8 As shown, n source threads correspond to n queues. Each source thread stores its corresponding slice serialized data at the head of the queue corresponding to the source thread, so that each target thread can subsequently extract the slice serialized data from the tail of the corresponding queue.

[0111] Optionally, the server obtains the amount of slice data to be stored in the current source thread, the storage capacity of the queue corresponding to the current source thread, and the amount of slice data already stored in the current source thread. The server determines the storage capacity to be used based on the amount of slice data to be stored, the amount of stored slice data, and the data expansion coefficient. If the storage capacity to be used exceeds the storage capacity of the queue, it indicates that the storage of slice data to be stored in the current source thread will be blocked. If the storage capacity to be used does not exceed the storage capacity of the queue, the task of storing the slice data in the queue of the current source thread is executed normally.

[0112] Among them, in order to improve the concurrency of the storage data structure, the intermediate storage module uses the thread ID as the key and LinkedBlockingDeque as the value to form a key-value pair data structure located by the thread. When multiple source threads obtain slices and serialize them and write them into the queue concurrently, they will be written into the queue corresponding to the source thread.

[0113] In this embodiment, by concurrently executing serialization and writing to the queue corresponding to each source thread through multiple source threads, the concurrency capability can be improved, the efficiency of writing slice serialized data to the queue can be improved, and thus the processing efficiency of data migration can be improved.

[0114] In an exemplary embodiment, a method for performing full data migration by multi-threading on the target side is as follows: Figure 9As shown, it includes steps S902 to S904, and obtains serialized data of each slice from the intermediate storage according to the target slice size, including step S902, obtaining serialized data of each slice from the target queue through each target-end thread in multiple target-end threads; wherein the source-end thread corresponds to the target-end thread one-to-one, and the target queue represents the queue corresponding to the source-end thread and the target-end thread.

[0115] Optionally, each target-side thread obtains each slice serialization data from a target queue through each target-side thread in a plurality of target-side threads; wherein the source-side thread corresponds to the target-side thread one-to-one, and the target queue represents the queue corresponding to the source-side thread and the target-side thread. Figure 8 As shown, one source thread corresponds to one queue and one target thread. Therefore, the size of the source thread pool (i.e., the number of source threads) is equal to the number of queues, which is equal to the size of the target thread pool (i.e., the number of target threads). All threads execute concurrently without interfering with each other. It should be noted that because the target slice size is calculated based on the full number of threads, even if all threads execute concurrently at the same time, it will not exceed the server's available memory.

[0116] According to the slice serialization data, the target migration data is obtained, and the target migration data is written into the target end database, including step S904, performing a deserialization operation on each slice serialization data through each target end thread to obtain the target migration data corresponding to each target end thread, and writing the target migration data into the target end database through each target end thread.

[0117] Optionally, the server deserializes the serialized data of each slice through each target-side thread to obtain the target migration data corresponding to each target-side thread. Figure 8 As shown in the figure, n target-side threads on the server concurrently deserialize the serialized slice data obtained from the queue. Each target-side thread writes the target migration data to the target-side database in batches.

[0118] In this embodiment, by having multiple target-end threads concurrently read the slice serialized data of their respective target-end threads from the corresponding queue, the concurrency capability can be improved, the efficiency of writing data to the target can be improved, and thus the processing efficiency of data migration can be improved.

[0119] In an exemplary embodiment, the data migration scenario of MongoDB, a document-based database, is used as an example. The system architecture is as follows: Figure 10 As shown in the figure, the source MongoDB, source read module, intermediate storage module, and target write module. This method is divided into two phases: the first phase is the pre-configuration phase, and the second phase is the data migration phase.

[0120] Pre-configuration phase: Before performing a full data migration, the server obtains data migration configuration parameters, the average slice size of the database table, and the server's available memory. Data migration configuration parameters include the number of full threads, the number of copies, and the data expansion factor. The number of copies includes the number of source and target copies. For example, if the number of full threads is n, the number of source copies + the number of target copies + 1 = the total number of copies. The number of source and target copies is 3, the data expansion factor is 5, and the server performance parameters are 46C256GB, with 90% available memory. The maximum slice size = available memory / total number of copies / number of full threads / data expansion factor = 256GB × 1024 × 0.9 / (3 + 3 + 1) / n / 5. The number of full threads can be freely set based on the above parameters: the number of full threads = the number of source threads = the number of target threads. The server usually splits the table data into integer multiples of the average slice size AVchunkSize for higher efficiency, so the target slice size chunkSize is usually selected as an integer multiple between (average slice size AVchunkSize, maximum slice size maxChunkSize). The server writes the above configured data into the configuration file of the configuration module so that the corresponding data can be retrieved from the configuration file later. If n is 10, the maximum slice size maxChunkSize = 675MB; the target slice size chunkSize is selected as an integer multiple from (64MB, 675MB), such as 384MB.

[0121] Data migration phase: The server obtains the server's account and connection information for the source database and the target database through the configuration module, such as user name, password, IP address port, connection string, etc. Based on the account and connection information, establish a communication connection between the server and the source database and the target database. The server obtains the structural information of the source database; generates corresponding Data Definition Language (DDL) statements based on the structural information of the database; for each Data Definition Language DDL statement, executes each Data Definition Language DDL statement in the target database, and writes the structural information of the source database into the target database. If the database is a Mongo database, after establishing a connection to the source database through the MongoDB driver, information including indexes and table creation statements is obtained through MongoDB's built-in listIndexes and other commands. The server replays the structural information of the source database in the target database to complete the structural migration.

[0122] After completing the structure migration, data migration is also required. The server determines the database table that matches the data to be migrated, based on the hash value of the data to be migrated. It then queries the database table for the target slice size, such as 384MB as mentioned above. Each source thread corresponds to a slice thread, for a total of 10 source threads and 10 corresponding slice threads.

[0123] For each source thread, the source thread slices each database table according to the target slice size to obtain the range information of the database table; based on the range information of the database table corresponding to the source thread, the corresponding data to be migrated is read from each database table; the source thread serializes the data to be migrated to obtain the slice serialized data corresponding to each source thread; the intermediate storage includes multiple queues; the queues correspond one-to-one with the source threads; in the order of each database table, the source thread stores the corresponding slice serialized data in the queue corresponding to the source thread. 10 slicing threads concurrently slice each database table according to the target slice size to obtain the range information of the database table corresponding to each source thread; 10 source threads can concurrently read the data to be migrated, serialize it, and store it in the queue corresponding to the source thread.

[0124] For each target-side thread, the target thread retrieves each slice's serialized data from the target queue. Source threads correspond one-to-one with target threads, and the target queue represents the queue corresponding to each source and target thread. The target thread deserializes each slice's serialized data to obtain the target migration data corresponding to each target thread, and then writes the target migration data to the target database. Ten target-side threads concurrently retrieve each slice's serialized data from the target queue. Ten target-side threads concurrently deserialize each slice's serialized data to obtain the target migration data corresponding to each target thread, and then concurrently write the target migration data to the target database.

[0125] When using traditional import and export tools, if an exception occurs during the data migration process, data consistency and integrity cannot be ensured, and there is no migration progress record information. In the event of an abnormal interruption, it is impossible to recover from the breakpoint and usually the only option is to clean up the target end and re-import.

[0126] This embodiment further includes: after the target-side thread writes the target migration data into the target-side database, recording progress information of the target migration data, where the progress information includes fields such as library, table, and completion status.

[0127] If the data migration process is interrupted, the server restarts the data migration program and reads the progress information. Based on the progress information, the source thread retrieves the serialized data of the incomplete shards and writes it to the queue corresponding to the source thread. The target thread reads the serialized data of the incomplete shards and deserializes it to obtain the incomplete target migration data. The target thread writes the incomplete target migration data to the target database, changes the status, and records the corresponding progress information. This allows for recovery from interruptions without having to clean up the target and start over, ensuring data migration consistency.

[0128] It's important to note that all source threads run concurrently without interfering with each other, and all target threads run concurrently without interfering with each other. Each target thread corresponding to a source thread belongs to the same process. This implements a parameterizable, performance-adaptive data migration process, avoiding performance issues with large data volumes, such as excessive source database resource usage and migration program OOM errors, thereby enhancing the robustness of the data migration process.

[0129] This method is used for performance testing. Without this method, traditional tools such as mongodump and mongostore typically have average speeds between 50MB / s and 200MB / s.

[0130] When using this method, the full performance of this method was tested on a 46C256G physical machine, set to multi-threaded mode. The reference data is shown in the table:

[0131] Table 1 Reference data

[0132]

[0133]

[0134] This method is faster than traditional tools overall and will not cause data migration interruptions due to exceptions.

[0135] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0136] Based on the same inventive concept, embodiments of the present application also provide a data migration device for implementing the aforementioned data migration method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following data migration device embodiments can be found in the aforementioned limitations on the data migration method and are not further elaborated here.

[0137] In an exemplary embodiment, Figure 11 As shown, a data migration device is provided, including: a source-side reading module 1101 and a target-side writing module 1102, wherein:

[0138] The source-side reading module 1101 is used to obtain each database table of the source-side database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and the available memory of the server; it is also used to slice each database table according to the target slice size to obtain slice information; and it is also used to obtain target migration data based on the slice information.

[0139] The target-side writing module 1102 is configured to write the target migration data into the target-side database.

[0140] In an exemplary embodiment, data migration configuration parameters include the number of full threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the number of copies on the source side and the number of copies on the target side; and the method for determining the target slice size includes:

[0141] The configuration module is used to determine the total number of copies based on the number of copies on the source side and the number of copies on the target side; determine the maximum slice size based on the server's available memory, number of full threads, data expansion coefficient and total number of copies; and determine the target slice size based on the average slice size and maximum slice size.

[0142] In an exemplary embodiment, the source-end reading module 1101 is further configured to obtain slice serialization data according to the slice information, and store the slice serialization data in an intermediate storage.

[0143] The target-side writing module 1102 is further configured to obtain serialized data of each slice from the intermediate storage according to the target slice size; obtain target migration data according to the slice serialized data, and write the target migration data into the target-side database.

[0144] In an exemplary embodiment, the database table carries data to be migrated; the source-end reading module 1101 is also used to read the corresponding data to be migrated from the database according to the slice information; serialize the data to be migrated to obtain slice serialized data; and store each slice serialized data in the intermediate storage in sequence.

[0145] In an exemplary embodiment, the slice information includes range information of the database table; the intermediate storage includes multiple queues; the source-end reading module 1101 is also used to read the corresponding data to be migrated from each database table through each source-end thread in multiple source-end threads according to the range information of the corresponding database table; serialize the data to be migrated through each source-end thread to obtain the slice serialized data corresponding to each source-end thread; and store the corresponding slice serialized data in the queue corresponding to the source thread through each source-end thread in the order of each database table.

[0146] In an exemplary embodiment, the target-side writing module 1102 is further used to obtain serialized data of each slice from a target queue through each target-side thread among multiple target-side threads; wherein the source-side thread corresponds to the target-side thread one-to-one, and the target queue represents the queue corresponding to the source-side thread and the target-side thread; deserialize the serialized data of each slice through each target-side thread to obtain the target migration data corresponding to each target-side thread, and write the target migration data into the target-side database through each target-side thread.

[0147] Each module in the aforementioned data migration device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0148] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 12As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The computer device is used to store migration data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration method is implemented.

[0149] Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0150] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0153] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile memory and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable logic unit (PLC), a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, and the like.

[0154] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0155] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data migration method, characterized in that: The method comprises: Obtain each database table of the source database, and query the target slice size of the database table; the target slice size is determined according to the average slice size of the database table, data migration configuration parameters, and available memory of the server; Slicing each of the database tables according to the target slice size to obtain slice information; The target migration data is obtained according to the slice information, and the target migration data is written into the target end database.

2. The method according to claim 1, characterized in that The data migration configuration parameters include the number of full threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the number of copies at the source end and the number of copies at the target end; the method for determining the size of the target slice includes: Determining the total number of copies according to the number of copies at the source end and the number of copies at the target end; Determining a maximum slice size based on available memory of the server, the number of full threads, the data expansion factor, and the total number of copies; A target slice size is determined according to the average slice size and the maximum slice size.

3. The method according to claim 1, characterized in that: The obtaining target migration data according to the slice information and writing the target migration data into a target-side database includes: Obtaining slice serialization data according to the slice information, and storing the slice serialization data in intermediate storage; According to the target slice size, acquiring serialized data of each slice from the intermediate storage; According to the slice serialization data, target migration data is obtained, and the target migration data is written into the target end database.

4. The method according to claim 3, characterized in that The database table carries data to be migrated; the obtaining slice serialization data according to the slice information, and storing the slice serialization data in intermediate storage, includes: Reading corresponding data to be migrated from the database according to the slice information; Performing a serialization operation on the data to be migrated to obtain slice serialization data; Each of the slice serialized data is stored in the intermediate storage in sequence.

5. The method according to claim 4, characterized in that The slice information includes range information of a database table; and reading corresponding data to be migrated from the database according to the slice information includes: Reading corresponding data to be migrated from each database table by each of the source threads in the multiple source threads according to the range information of the corresponding database table; The serializing the data to be migrated to obtain slice serialized data includes: Performing a serialization operation on the data to be migrated through each of the source-end threads to obtain slice serialization data corresponding to each of the source-end threads; The intermediate storage includes a plurality of queues; the queues correspond to the source threads one by one; and storing each of the slice serialized data in the intermediate storage in sequence includes: According to the order of each database table, each source-end thread stores the corresponding slice serialized data in the queue corresponding to the source-end thread.

6. The method according to claim 5, characterized in that The acquiring, according to the target slice size, the serialized data of each slice from the intermediate storage comprises: Obtaining each of the slice serialization data from a target queue through each of the multiple target threads; wherein the source thread corresponds to the target thread one-to-one, and the target queue represents a queue corresponding to the source thread and the target thread; The step of obtaining target migration data according to the slice serialization data and writing the target migration data into a target end database includes: Deserialization operation is performed on each slice serialized data through each target-end thread to obtain the target migration data corresponding to each target-end thread, and the target migration data is written into the target-end database through each target-end thread.

7. A data migration device, characterized in that: The device comprises: The source end reading module is used to obtain each database table of the source end database and query the target slice size of the database table; the target slice size is determined according to the average slice size of the database table, the data migration configuration parameters and the available memory of the server; It is also used to slice each of the database tables according to the target slice size to obtain slice information; Also used to obtain target migration data according to the slice information; The target-side writing module is used to write the target migration data into the target-side database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images