Data migration method and device, computer device, readable storage medium and program product

Abstract

The application relates to a data migration method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring each database table of a source end database, and querying a target slice size of the database table; the target slice size is determined according to an average slice size of the database table, a data migration configuration parameter and available memory of a server; each database table is sliced according to the target slice size to obtain slice information; and target migration data is obtained according to the slice information, and the target migration data is written into a target end database. The method can avoid performance problems such as insufficient storage space or abnormal exit, thereby improving the robustness of a data migration program.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of big data technology, the demand for database stability and availability is increasing day by day. Data migration technology has emerged to meet the needs of scenarios such as cloud migration, backup and disaster recovery, scaling up and down, archiving, and data analysis.

[0003] In traditional techniques, data migration is carried out using conventional export and import tools such as mongoexport, mongoimport, mongodump, and mongostore.

[0004] However, since the server's performance parameters are fixed, large-scale data volumes can lead to performance issues such as insufficient storage space or file system performance limitations causing abnormal exits. Summary of the Invention

[0005] Therefore, it is necessary to provide a data migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can avoid performance problems such as insufficient storage space or abnormal exit, and improve the robustness of data migration programs, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a data migration method, the method comprising:

[0007] Obtain each database table from the source database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and the available memory of the server.

[0008] Based on the target slice size, each of the database tables is sliced ​​to obtain slice information;

[0009] The target migration data is obtained based on the slice information, and the target migration data is written into the target database.

[0010] In one embodiment, the data migration configuration parameters include the maximum number of threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the source copy number and the target copy number; the method for determining the target slice size includes:

[0011] The total number of copies is determined based on the source copy number and the target copy number;

[0012] The maximum slice size is determined based on the server's available memory, the maximum number of threads, the data expansion coefficient, and the total number of copies.

[0013] The target slice size is determined based on the average slice size and the maximum slice size.

[0014] In one embodiment, obtaining target migration data based on the slice information and writing the target migration data into the target database includes:

[0015] Based on the slice information, slice serialized data is obtained and stored in intermediate storage;

[0016] Based on the target slice size, retrieve the serialized data of each slice from the intermediate storage;

[0017] Based on the sliced ​​serialized data, the target migration data is obtained and written into the target database.

[0018] In one embodiment, the database table carries data to be migrated; the step of obtaining slice serialized data based on the slice information and storing the slice serialized data in intermediate storage includes:

[0019] The corresponding data to be migrated is read from the database based on the slice information;

[0020] The data to be migrated is serialized to obtain sliced ​​serialized data;

[0021] The serialized data of each slice is stored in the intermediate storage in sequence.

[0022] In one embodiment, the slice information includes range information of a database table; the step of reading the corresponding data to be migrated from the database based on the slice information includes:

[0023] Each of the multiple source threads reads the corresponding data to be migrated from each of the database tables according to the range information of their respective database tables.

[0024] The step of serializing the data to be migrated to obtain sliced ​​serialized data includes:

[0025] The data to be migrated is serialized by each of the source threads to obtain sliced ​​serialized data corresponding to each source thread.

[0026] The intermediate storage includes multiple queues; each queue corresponds one-to-one with a source thread; the step of storing the serialized data of each slice into the intermediate storage in sequence includes:

[0027] According to the order of each database table, the corresponding slice serialized data is stored in the queue corresponding to each source thread through each source thread.

[0028] In one embodiment, retrieving the serialized data of each slice from the intermediate storage according to the target slice size includes:

[0029] Each of the multiple target threads retrieves the serialized slice data from the target queue; wherein, the source thread corresponds one-to-one with the target thread, and the target queue represents the queue corresponding to the source thread and the target thread;

[0030] The step of obtaining target migration data based on the sliced ​​serialized data and writing the target migration data into the target end database includes:

[0031] Each target thread performs deserialization operations on each slice serialized data to obtain the target migration data corresponding to each target thread, and then writes the target migration data into the target database through each target thread.

[0032] Secondly, this application also provides a data migration apparatus, the apparatus comprising:

[0033] The source-side reading module is used to obtain each database table of the source database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and the available memory of the server.

[0034] It is also used to slice each of the database tables according to the target slice size to obtain slice information;

[0035] It is also used to obtain target migration data based on the slice information;

[0036] The target end writing module is used to write the target migration data into the target end database.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0040] The aforementioned data migration methods, apparatus, computer equipment, computer-readable storage media, and computer program products determine the target slice size before data migration based on the average slice size of the database tables, data migration configuration parameters, and the available memory of the server. In other words, the target slice size for data migration is determined through server performance parameters. By querying the target slice size of the database tables, each database table is sliced ​​according to the target slice size. The target migration data is obtained based on the slice information and written to the target database. Since server performance parameters have been considered when slicing each database table, abnormal interruptions due to insufficient server performance parameters during data migration are avoided. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a diagram illustrating the application environment of a data migration method in one embodiment.

[0043] Figure 2 This is a flowchart illustrating a data migration method in one embodiment;

[0044] Figure 3 A flowchart illustrating how the target slice size is determined in one embodiment;

[0045] Figure 4 This is a schematic diagram illustrating the process of writing target migration data into the target database in one embodiment;

[0046] Figure 5 This is a schematic diagram illustrating the process of storing sliced ​​serialized data to intermediate storage in one embodiment;

[0047] Figure 6 This is a schematic diagram of the modules included in a server that performs a data migration method in one embodiment;

[0048] Figure 7 This is a flowchart illustrating a method for full data migration using multiple threads on the source side in one embodiment.

[0049] Figure 8This is a flowchart illustrating a method for full data migration using multiple threads on the source and target sides in one embodiment.

[0050] Figure 9 This is a flowchart illustrating a method for full data migration using multi-threaded approaches on the target side in one embodiment.

[0051] Figure 10 This is a schematic diagram of the system architecture for one embodiment;

[0052] Figure 11 This is a structural block diagram of a data migration device in one embodiment;

[0053] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The data migration method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the source database 106 communicates with the server 102 via a network. The source database 106 stores the data to be migrated. The source database 106 can be integrated onto the server 102, or it can be located in the cloud or on another network server. The source database 106 can be a single source database or multiple source databases. The target database 104 communicates with the server 102 via a network. The target database 104 stores the migrated data. The target database 104 can be integrated onto the server 102, or it can be located in the cloud or on another network server. The server 102 retrieves the database tables from the source database and queries the target slice size of the database tables. The target slice size is determined based on the average slice size of the database tables, data migration configuration parameters, and the available memory of the server. Based on the target slice size, each database table is sliced ​​to obtain slice information. Based on the slice information, the target migration data is obtained and written to the target database. The server 102 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0056] In one exemplary embodiment, such as Figure 2 As shown, a data migration method is provided, which is applied to... Figure 1 Taking the server in the example, the explanation includes the following steps S202 to S206. Wherein:

[0057] Step S202: Obtain each database table from the source database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and the available memory of the server.

[0058] Data migration refers to copying the structural information and data of objects from one database system to another. Full migration refers to the complete migration of all data from the source database to the target database at a given moment, including all table structures, records, indexes, etc. The source database is the one sending the data during the migration process. The target database is the one receiving the data during the migration process. The target slice size is a granular representation of a database table. The unit of target slice size is MB, to avoid excessively large individual slices causing Out Of Memory (OOM) errors, i.e., exceeding the memory limit.

[0059] Optionally, for example in a full migration scenario, before obtaining the database tables from the source database (i.e., before migrating the data), a structural migration is performed first. Structural migration refers to migrating the structural information of database objects, such as tables, views, functions, and stored procedures. The server obtains the account and connection information for the source and target databases through a configuration module, such as username, password, IP address, port, and connection string. Based on the account and connection information, a communication connection is established between the server and the source and target databases. The server obtains the structural information of the source database; it generates corresponding Data Definition Language (DDL) statements from this information; and for each DDL statement, it executes it in the target database, writing the structural information from the source database into the target database. If the database is a MongoDB database, after establishing a connection to the source database through the MongoDB driver, information including indexes and table creation statements is obtained using MongoDB's built-in `listIndexes` commands. The server then replays the structural information from the source database in the target database, completing the structural migration.

[0060] Optionally, after the structure migration is completed, data migration is also required. The server obtains each database table from the source database, including: obtaining the data to be migrated from the source database; and obtaining each database table that matches the data to be migrated.

[0061] In one optional embodiment, the server determines the database tables that each source thread retrieves based on the hash value of the data to be migrated. For example, if there are 100 database tables in the source database that match the data to be migrated, the server retrieves all 100 database tables.

[0062] In one optional embodiment, each of the multiple source threads on the server determines which database table matches the data to be migrated based on the hash value of the data to be migrated. At any given time, only one source thread executes the task of retrieving the database table. For example, if there are 100 database tables in the source database that match the data to be migrated, and 10 threads exist on the server, thread 1 retrieves tables 1, 3, 6...100; thread 2 retrieves tables 2, 5, 7...99; ... thread 10 retrieves tables xx...xx. When thread 1 retrieves database table 1 in the source database, thread 1 is locked, preventing other threads from retrieving the corresponding database table from the source database, thus preventing multiple threads from retrieving the same database table simultaneously.

[0063] Optionally, the server retrieves each database table from the source database and queries the target slice size for each table. The target slice size is determined based on the average slice size of the database tables, data migration configuration parameters, and the available memory on the server.

[0064] Step S204: Slice each database table according to the target slice size to obtain slice information.

[0065] In this context, a slice refers to a segment of data in a database table where start and end points are defined, allowing the retrieval of data from those points. Granularity controls the degree of control over the slice size.

[0066] Optionally, the server slices each database table according to the target slice size to obtain slice information, where the slice information is the start and end point information of the database table, that is, the slice information is a range information.

[0067] Optionally, when multiple source threads exist, each source thread pool includes a slicing thread. For each source read thread's corresponding slicing thread, the database tables are sliced ​​according to the target slice size to obtain the slice information for each source thread. The number of source threads is configured through a configuration module. It is understood that performance parameters such as memory usage, thread concurrency, and slice size must be pre-configured in the configuration file.

[0068] Step S206: Obtain the target migration data based on the slice information and write the target migration data into the target database.

[0069] Optionally, the server processes the slice information to obtain the target migration data and writes it to the target database. For example, the server obtains the corresponding data to be migrated based on the slice information, and obtains the target migration data by serializing and deserializing the data to be migrated. The server then writes the target migration data to the target database in batches.

[0070] In the data migration method described above, before data migration, the target slice size is determined based on the average slice size of the database tables, data migration configuration parameters, and the available memory of the server. In other words, the target slice size for data migration is determined through server performance parameters. By querying the target slice size of the database tables, each database table is sliced ​​according to the target slice size. The target migration data is obtained based on the slice information and written to the target database. Since server performance parameters have been considered when slicing each database table, abnormal interruptions due to insufficient server performance parameters during data migration are avoided.

[0071] In one exemplary embodiment, such as Figure 3 As shown, the data migration configuration parameters include the maximum number of threads, the number of copies, and the data expansion coefficient. The number of copies includes the source copy number and the target copy number. The method for determining the target slice size includes steps S302 to S306. Wherein:

[0072] Step S302: Determine the total number of copies based on the number of copies at the source end and the number of copies at the target end.

[0073] The maximum number of threads refers to the maximum number of threads the server can run. Source copy count refers to the number of copies made during the process of reading sliced ​​serialized data from the source database and storing it. For example, database queries, serialization, and data processing all generate copies. Target copy count refers to the number of copies made during the process of reading sliced ​​serialized data from storage and writing it to the target database, such as during data processing operations like deserialization. Typically, the source or target copy count is around 3. The data bloat factor refers to the size of the serialized instance object after data processing (e.g., the document data stored in MongoDB may have a multi-level nested structure) compared to the original text data. This is typically set between 5 and 10 times larger.

[0074] Optionally, in an architecture where the source database connects to intermediate storage and the intermediate storage connects to the target database, the total number of copies equals the number of copies at the source, the number of copies at the target, and one copy at the intermediate storage.

[0075] Step S304: Determine the maximum slice size based on the server's available memory, maximum number of threads, data expansion coefficient, and total number of copies.

[0076] Optionally, after obtaining the specific values ​​of its available memory, configured maximum number of threads, data expansion coefficient, and total number of copies, the server can substitute them into the first relational expression. The first relational expression is determined based on the second relational expression.

[0077] The second relation is: the maximum memory occupied by the program at a certain moment is approximately equal to the source end's full thread count x {maximum slice size maxChunkSize x expansion coefficient x (source end copy count + 1)} + the target end's full thread count x (maximum slice size maxChunkSize x expansion coefficient x target end copy count). In the architecture where the source database connects to intermediate storage, and intermediate storage connects to the target database, the source end's full thread count = the target end's full thread count = the total number of threads. Translating the second relation, the maximum memory occupied by the program at a certain moment is approximately equal to the full thread count × maximum slice size maxChunkSize × expansion coefficient × (source end copy count + 1 + target end copy count). The server determines the first relation based on the second relation: maximum memory / total number of copies / expansion coefficient / full thread count is approximately equal to the maximum slice size maxChunkSize. For example, taking the smallest 1C4G specification machine as an example, the expansion coefficient is 5, the source end copy count and the target end copy count are 3, and the full thread count is 1. If the configuration is set to use 90% of the memory, then the maximum MaxChunkSize = 4G x 1024x0.9 / (3+1+3) / 5 = 105MB.

[0078] Step S306: Determine the target slice size based on the average slice size and the maximum slice size.

[0079] In one database, the average slice size AVchunkSize is 64MB.

[0080] Alternatively, servers typically split table data more efficiently in multiples of AVchunkSize, so the target slice size (chunkSize) is usually chosen as an integer multiple between (AVchunkSize, maxChunkSize). If the server has limited available memory, the average slice size, such as 64MB, can be specified directly.

[0081] In this embodiment, the target slice size can be determined by configuring data migration parameters and the average slice size. On the one hand, splitting each database table according to the target slice size can avoid data migration interruption when performance is insufficient. On the other hand, splitting each database table according to the target slice size is more efficient.

[0082] In one exemplary embodiment, such as Figure 4As shown, the target migration data is obtained based on the slice information, and the target migration data is written into the target end database, including steps S402 to S406. Wherein:

[0083] Step S402: Obtain slice serialized data based on slice information, and store the slice serialized data in intermediate storage.

[0084] Optionally, the server reads the data to be migrated based on the slice information, serializes the data to be migrated, and obtains slice serialized data. The slice serialized data is then stored in intermediate storage in a first-in-first-out manner.

[0085] Step S404: Based on the target slice size, retrieve the serialized data of each slice from the intermediate storage.

[0086] Optionally, the server may pull multiple batches of slice serialization data from the tail of the intermediate storage queue, depending on the target slice size.

[0087] Step S406: Obtain target migration data based on slice serialization data, and write the target migration data into the target database.

[0088] Optionally, the server performs deserialization on the retrieved slice serialized data. If the database is MongoDB, the server converts the field description metadata and binary stream values ​​back into the MongoDB-recognizable BsonDocument type to obtain the target migration data.

[0089] In this embodiment, slice serialization data is obtained based on slice information. The slice serialization data is first stored in intermediate storage for subsequent writing by the target end, thereby decoupling source end reading and serialization from target end writing and further improving performance controllability.

[0090] In one exemplary embodiment, such as Figure 5 As shown, the database table carries data to be migrated; based on the slice information, slice serialized data is obtained, and the slice serialized data is stored in intermediate storage, including steps S502 to S506. Wherein:

[0091] Step S502: Read the corresponding data to be migrated from the database table based on the slice information.

[0092] Before reading the corresponding data to be migrated from the database based on the slice information, the server includes a configuration module, a scheduling module, a structure migration module, a source read module, an intermediate storage module, and a target read / write module, such as... Figure 6As shown in the diagram. The configuration module is used for configuring source and target database information, migration objects, intermediate storage, and performance parameters. The server loads settings required by other modules from the configuration file through the configuration module. This includes account and connection information for the migration module's source and target databases, such as username, password, IP address, port, and connection string. Performance-related parameters for the migration module include memory usage, thread concurrency, batch size, and slice size. The intermediate storage configuration allows specifying whether to use memory, the local file system, or implement interfaces to extend to other storage formats. The migration object is a simplified data model that the server migrates from the source database according to instructions. It is specified to specific database table columns using JSON format and wildcards, and can also configure filtering rules and name mappings.

[0093] The scheduling module is responsible for loading configurations, sequentially scheduling structure migrations, starting the source-side reading module and the target-side writing module, and managing the lifecycle of each module. The program terminates when the source-side reading module has finished reading all data and the target-side writing module has finished writing all data. It is also responsible for writing progress information and handling logic for handling exception interruptions.

[0094] The structure migration module retrieves source database information from the configuration module. After establishing a database connection via the MongoDB driver, it uses built-in MongoDB commands such as `listIndexes` to obtain information including indexes and table creation statements. For MongoDB sharded sets, it also queries the `config` database for the shard key information matching the migration object. After reading the source database structure information, it records it as DDL statements and replays them on the target database.

[0095] The source-side reading module is used to obtain source database information and migration objects from the configuration module. For a replica set, after establishing a database connection through the MongoDB driver, it queries the database for information such as the database tables that match the migration objects, and performs split queries at the granularity of the tables (collections) to be migrated.

[0096] The intermediate storage module is used to store fragmented serialized data and decouples the source-end read module from the target-end write module.

[0097] The target-side write module is used to obtain concurrent configuration parameters from the configuration module, and then uses multi-threading to concurrently pull serialized batches of fragmented data from the tail of the bidirectional queue of the intermediate storage module.

[0098] Optionally, the server reads the corresponding data to be migrated from each database based on the slice information.

[0099] Step S504: Serialize the data to be migrated to obtain sliced ​​serialized data.

[0100] Optionally, the server will serialize the migrated data to ensure compatibility with different database table structures. The intermediate data model for serialization includes field descriptions and actual data. The field descriptions mainly include metadata information such as primary key (_id), unique key, and binary stream type. Correspondingly, the actual data needs to store the specific values ​​of the primary key, unique key, and text converted into a binary stream corresponding to a database record.

[0101] Taking MongoDB as an example, the server uses BsonDocumentCodec to convert BsonDocument into a binary stream array byte[], and then encapsulates it through ByteBuffer to obtain sliced ​​serialized data.

[0102] Step S506: Serialize each slice of data and store it in the intermediate storage in sequence.

[0103] Optionally, each slice of serialized data is stored in intermediate storage sequentially. The intermediate storage primarily uses a thread-safe ConcurrentMap and a doubly blocking queue concurrent data structure to store the intermediate format data. The source end pushes data to the head of the queue, and the target end writes data by drawing data from the tail of the queue.

[0104] In this embodiment, data serialization is used to ensure compatibility with different database table structures. By storing the serialized data of each slice in intermediate storage, decoupling from the subsequent writing process is achieved.

[0105] In one exemplary embodiment, a method for performing full data migration using multiple threads on the source side, such as... Figure 7 As shown, steps S702 to S706 are included. The slicing information includes the range information of the database table. The corresponding data to be migrated is read from the database according to the slicing information. Step S702 involves each source thread in multiple source threads reading the corresponding data to be migrated from each database table according to the range information of its respective database table.

[0106] Optionally, the server uses multiple source threads, each of which reads the corresponding data to be migrated from its respective database table based on the range information of that database table. For example... Figure 8 As shown, the server uses n source threads to concurrently read the corresponding data to be migrated from each database table according to the range information of the database table.

[0107] The data to be migrated is serialized to obtain sliced ​​serialized data, including step S704, in which the data to be migrated is serialized through each source thread to obtain sliced ​​serialized data corresponding to each source thread.

[0108] Optionally, multiple source threads perform serialization operations on the data to be migrated, obtaining sliced ​​serialized data corresponding to each source thread. For example... Figure 8 As shown, the server has n source threads that perform serialization operations on the data to be migrated, resulting in sliced ​​serialized data corresponding to each source thread.

[0109] The intermediate storage includes multiple queues; each queue corresponds one-to-one with a source thread; the serialized data of each slice is stored in the intermediate storage in sequence, including step S706: according to the order of each database table, each source thread stores its corresponding serialized slice data into the queue corresponding to the source thread.

[0110] Optionally, the server stores its respective slice serialized data into the queue corresponding to each source thread. For example... Figure 8 As shown, n source threads correspond to n queues. Each source thread stores its corresponding slice serialized data at the head of its corresponding queue. This allows subsequent target threads to extract slice serialized data from the tail of their respective queues.

[0111] Optionally, the server obtains the amount of data to be stored in the current source thread, the storage capacity of the queue corresponding to the current source thread, and the amount of data already stored in the current source thread. The server determines the storage capacity to be used based on the amount of data to be stored, the amount of data already stored, and the data expansion factor. If the storage capacity to be used exceeds the queue's storage capacity, it indicates that the data to be stored in the current source thread will be blocked. If the storage capacity to be used does not exceed the queue's storage capacity, the task of storing data to the queue in the current source thread will execute normally.

[0112] To improve the concurrency of the stored data structure, the intermediate storage module uses thread ID as the key and LinkedBlockingDeque as the value, forming a key-value pair data structure that uses thread location. When multiple source threads obtain slices, serialize them, and concurrently write them to the queue, the data will be written to the queue corresponding to the source thread.

[0113] In this embodiment, by having multiple source threads concurrently perform serialization and write to the queue corresponding to each source thread, the concurrency capability can be improved, the efficiency of writing sliced ​​serialized data to the queue can be increased, and thus the processing efficiency of data migration can be improved.

[0114] In an exemplary embodiment, the method for multi-threaded full data migration on the target end is as follows: Figure 9As shown, steps S902 to S904 are included, in which serialized data of each slice is obtained from intermediate storage according to the target slice size. Step S902 involves each target thread in multiple target threads obtaining serialized data of each slice from the target queue. The source thread and the target thread correspond one-to-one, and the target queue represents the queue corresponding to the source thread and the target thread.

[0115] Optionally, each slice of serialized data is obtained from the target queue by each target thread among multiple target threads; wherein, there is a one-to-one correspondence between the source thread and the target thread, and the target queue represents the queue corresponding to the source thread and the target thread. Figure 8 As shown, one source thread corresponds to one queue and one target thread. Therefore, the size of the source thread pool, i.e., the number of source threads, equals the number of queues, which in turn equals the size of the target thread pool, i.e., the number of target threads. All threads process concurrently without interfering with each other. It should be noted that since the target slice size is calculated based on the full number of threads, even if all threads run concurrently at the same time, it will not exceed the available memory of the server.

[0116] Based on the slice serialization data, target migration data is obtained and written to the target end database. This includes step S904, where each target end thread performs deserialization operation on each slice serialization data to obtain target migration data corresponding to each target end thread, and writes the target migration data to the target end database through each target end thread.

[0117] Optionally, the server performs deserialization operations on the serialized data of each slice through each target thread to obtain the target migration data corresponding to each target thread. For example... Figure 8 As shown, n target-side threads on the server concurrently deserialize the sliced ​​serialized data they retrieve from the queue. The target migration data is then written to the target database in batches by each target-side thread.

[0118] In this embodiment, by having multiple target threads concurrently read the sliced ​​serialized data of their respective target threads from the corresponding queues, concurrency capabilities can be improved, the efficiency of writing data to the target can be increased, and thus the processing efficiency of data migration can be improved.

[0119] In an exemplary embodiment, a data migration scenario using MongoDB, a document-oriented database, is illustrated. The system architecture is as follows: Figure 10 As shown. The system consists of a source MongoDB module, a source read module, an intermediate storage module, and a target write module. This method is divided into two phases: the first phase is the pre-configuration phase, and the second phase is the data migration phase.

[0120] Pre-configuration phase: Before performing a full data migration, the server obtains data migration configuration parameters, the average slice size of the database tables, and the server's available memory. These parameters include the maximum number of threads, the number of copies, and the data expansion factor. The number of copies includes the source copy count and the target copy count. For example, if the maximum number of threads is n, then the total number of copies is: source copy count + target copy count + 1. Here, the source copy count and target copy count are both 3, the data expansion factor is 5, and the server's performance parameters are 46 cores and 256GB of RAM, with 90% available memory. The maximum slice size = available memory / total number of copies / maximum number of threads / data expansion factor = 256GB × 1024 × 0.9 / (3 + 3 + 1) / n / 5. The maximum number of threads can be freely set based on the above parameters, where the maximum number of threads = source thread count = target thread count. Servers typically partition table data more efficiently by using integer multiples of the average slice size AVchunkSize. Therefore, the target slice size chunkSize is usually chosen as an integer multiple between (average slice size AVchunkSize and maximum slice size maxChunkSize). The server writes the configured data to the configuration module's configuration file so that the corresponding data can be retrieved from the configuration file later. If n is 10, then the maximum slice size maxChunkSize = 675MB; the target slice size chunkSize is chosen as an integer multiple between (64MB and 675MB), such as 384MB.

[0121] Data migration phase: The server obtains the account and connection information for the source and target databases, such as username, password, IP address, port, and connection string, through the configuration module. Based on the account and connection information, it establishes communication connections between the server and the source and target databases. The server obtains the structure information of the source database; it generates corresponding Data Definition Language (DDL) statements from this structure information; for each DDL statement, it executes it in the target database, writing the structure information from the source database into the target database. If the database is a MongoDB database, after establishing a connection to the source database through the MongoDB driver, it uses built-in MongoDB commands such as `listIndexes` to obtain information including indexes and table creation statements. The server then replays the structure information from the source database in the target database, completing the structure migration.

[0122] After the structure migration is completed, data migration is also required. The server determines the database table that matches the data to be migrated based on the hash value of the data to be migrated, and queries the target slice size of the database table, such as 384MB as mentioned above; each source thread corresponds to one slicing thread, and there are a total of 10 source threads and 10 corresponding slicing threads.

[0123] For each source thread, the source thread slices each database table according to the target slice size to obtain the range information of the database table; based on the range information of the database table corresponding to the source thread, it reads the corresponding data to be migrated from each database table; the source thread performs a serialization operation on the data to be migrated to obtain the slice serialized data corresponding to each source thread; the intermediate storage includes multiple queues; each queue corresponds one-to-one with a source thread; according to the order of each database table, the source thread stores its corresponding slice serialized data into the queue corresponding to the source thread. Ten slicing threads concurrently slice each database table according to the target slice size to obtain the range information of the database table corresponding to each source thread; the ten source threads can concurrently read the data to be migrated, perform serialization operations, and store it into the queue corresponding to the source thread.

[0124] For each target thread, it retrieves the serialized data of each slice from the target queue. There is a one-to-one correspondence between the source and target threads, and the target queue represents the queues corresponding to the source and target threads. The target thread performs deserialization on the serialized data of each slice to obtain the target migration data for each target thread, and then writes the target migration data into the target database. Ten target threads concurrently retrieve the serialized data of each slice from the target queue; ten target threads concurrently perform deserialization on the serialized data of each slice to obtain the target migration data for each target thread; and ten target threads concurrently write the target migration data into the target database.

[0125] Traditional import / export tools cannot guarantee data consistency and integrity if an anomaly occurs during data migration, and there is no migration progress information. If an anomaly occurs, it is impossible to recover from the breakpoint, and usually the only solution is to clean the target end and re-import.

[0126] This embodiment also includes: after the target migration data is written to the target database by the target thread, the progress information of the target migration data is recorded, including fields such as database, table, and completion status.

[0127] When the data migration process is abnormally interrupted, the data migration program on the server is restarted and the progress information is read. Based on the progress information, the source thread retrieves the fragmented serialized data with an incomplete status. This incomplete fragmented serialized data is written to the queue corresponding to the source thread. The target thread reads the incomplete fragmented serialized data and performs deserialization to obtain the incomplete target migration data. The target thread writes the incomplete target migration data to the target database, changes its status, and records the corresponding progress information. This achieves abnormal interruption recovery without needing to clean up the target end and start over, ensuring the consistency of the data migration.

[0128] It should be noted that all source threads process concurrently without interfering with each other, and all target threads process concurrently without interfering with each other. Each source thread and its corresponding target thread belong to the same process. This implements a parameterized, configurable, performance-adaptive data migration process, avoiding performance issues such as excessive source database resource consumption and Out-of-Memory (OOM) errors in large-scale data migrations, thus enhancing the robustness of the data migration program.

[0129] Use this method for performance testing. Without this method, the average speed of traditional tools such as mongodump and mongostore is typically between 50MB / s and 200MB / s.

[0130] Using this method, its full performance was tested on a 46C256G physical machine, set to multi-threaded mode. Reference data is shown in the table below.

[0131] Table 1 Reference Data

[0132]

[0133]

[0134] Using this method is generally faster than traditional tools, and it will not cause any anomalies that could interrupt the data migration process.

[0135] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0136] Based on the same inventive concept, this application also provides a data migration apparatus for implementing the data migration method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data migration apparatus embodiments provided below can be found in the limitations of the data migration method described above, and will not be repeated here.

[0137] In one exemplary embodiment, such as Figure 11 As shown, a data migration device is provided, including: a source-end reading module 1101 and a target-end writing module 1102, wherein:

[0138] The source-side reading module 1101 is used to obtain each database table of the source database and query the target slice size of the database table. The target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and available memory of the server. It is also used to slice each database table according to the target slice size to obtain slice information. It is also used to obtain the target migration data based on the slice information.

[0139] The target end writing module 1102 is used to write the target migration data into the target end database.

[0140] In an exemplary embodiment, the data migration configuration parameters include the maximum number of threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the source copy number and the target copy number; the method for determining the target slice size includes:

[0141] The configuration module is used to determine the total number of copies based on the number of copies at the source and the number of copies at the target; to determine the maximum slice size based on the server's available memory, maximum number of threads, data expansion coefficient, and total number of copies; and to determine the target slice size based on the average slice size and the maximum slice size.

[0142] In an exemplary embodiment, the source reading module 1101 is further configured to obtain slice serialized data based on the slice information and store the slice serialized data in intermediate storage.

[0143] The target end writing module 1102 is also used to obtain the serialized data of each slice from the intermediate storage according to the target slice size; obtain the target migration data according to the slice serialized data; and write the target migration data into the target end database.

[0144] In an exemplary embodiment, the database table carries data to be migrated; the source reading module 1101 is further configured to read the corresponding data to be migrated from the database according to the slice information; perform a serialization operation on the data to be migrated to obtain slice serialized data; and store each slice serialized data in the intermediate storage in sequence.

[0145] In an exemplary embodiment, the slice information includes the range information of the database tables; the intermediate storage includes multiple queues; the source-end reading module 1101 is further configured to read the corresponding data to be migrated from each database table according to the range information of the corresponding database table by each source-end thread among the multiple source-end threads; to perform serialization operation on the data to be migrated by each source-end thread to obtain the slice serialized data corresponding to each source-end thread; and to store the corresponding slice serialized data into the queue corresponding to each source-end thread according to the order of each database table.

[0146] In an exemplary embodiment, the target end writing module 1102 is further configured to obtain each slice serialized data from the target queue through each target end thread among multiple target end threads; wherein, the source end thread corresponds one-to-one with the target end thread, and the target queue represents the queue corresponding to the source end thread and the target end thread; each target end thread performs deserialization operation on each slice serialized data to obtain target migration data corresponding to each target end thread, and each target end thread writes the target migration data into the target end database.

[0147] Each module in the aforementioned data migration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0148] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The computer device stores migration data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data migration method.

[0149] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0150] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0151] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0152] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0153] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0155] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data migration method, characterized in that, The method includes: Obtain each database table from the source database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and the available memory of the server. Based on the target slice size, each of the database tables is sliced ​​to obtain slice information; The target migration data is obtained based on the slice information, and the target migration data is written into the target database. The data migration configuration parameters include the maximum number of threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the source copy number and the target copy number; the method for determining the target slice size includes: The total number of copies is determined based on the source copy number and the target copy number; The maximum slice size is determined based on the server's available memory, the maximum number of threads, the data expansion coefficient, and the total number of copies. The target slice size is determined based on the average slice size and the maximum slice size.

2. The method according to claim 1, characterized in that, The step of obtaining target migration data based on the slice information and writing the target migration data into the target database includes: Based on the slice information, slice serialized data is obtained and stored in intermediate storage; Based on the target slice size, retrieve the serialized data of each slice from the intermediate storage; Based on the sliced ​​serialized data, the target migration data is obtained and written into the target database.

3. The method according to claim 2, characterized in that, The database table carries data to be migrated; the step of obtaining slice serialized data based on the slice information and storing the slice serialized data in intermediate storage includes: The corresponding data to be migrated is read from the database based on the slice information; The data to be migrated is serialized to obtain sliced ​​serialized data; The serialized data of each slice is stored in the intermediate storage in sequence.

4. The method according to claim 3, characterized in that, The slice information includes the range information of the database table; the step of reading the corresponding data to be migrated from the database according to the slice information includes: Each of the multiple source threads reads the corresponding data to be migrated from each of the database tables according to the range information of their respective database tables. The step of serializing the data to be migrated to obtain sliced ​​serialized data includes: The data to be migrated is serialized by each of the source threads to obtain sliced ​​serialized data corresponding to each source thread. The intermediate storage includes multiple queues; each queue corresponds one-to-one with a source thread; the step of storing the serialized data of each slice into the intermediate storage in sequence includes: According to the order of each database table, the corresponding slice serialized data is stored in the queue corresponding to each source thread through each source thread.

5. The method according to claim 4, characterized in that, The step of retrieving the serialized data of each slice from the intermediate storage according to the target slice size includes: Each of the multiple target threads retrieves the serialized slice data from the target queue; wherein, the source thread corresponds one-to-one with the target thread, and the target queue represents the queue corresponding to the source thread and the target thread; The step of obtaining target migration data based on the sliced ​​serialized data and writing the target migration data into the target end database includes: Each target thread performs deserialization operations on each slice serialized data to obtain the target migration data corresponding to each target thread, and then writes the target migration data into the target database through each target thread.

6. A data migration device, characterized in that, The device includes: The source-side reading module is used to obtain each database table of the source database and query the target slice size of the database table; the target slice size is determined based on the average slice size of the database table, data migration configuration parameters, and the available memory of the server. It is also used to slice each of the database tables according to the target slice size to obtain slice information; It is also used to obtain target migration data based on the slice information; The target end writing module is used to write the target migration data into the target end database; The data migration configuration parameters include the maximum number of threads, the number of copies, and the data expansion coefficient, wherein the number of copies includes the source copy number and the target copy number; the configuration module is used to determine the total number of copies based on the source copy number and the target copy number; determine the maximum slice size based on the available memory of the server, the maximum number of threads, the data expansion coefficient, and the total number of copies; and determine the target slice size based on the average slice size and the maximum slice size.

7. The apparatus according to claim 6, characterized in that, The source-end reading module is further configured to obtain slice serialized data based on the slice information and store the slice serialized data in intermediate storage; The target end writing module is further configured to obtain the serialized data of each slice from the intermediate storage according to the target slice size; obtain the target migration data according to the slice serialized data; and write the target migration data into the target end database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images