Data migration method and device and computer equipment

Abstract

The invention relates to a data migration method and device and computer equipment. The method comprises the following steps: acquiring a plurality of first data storage units in a source database, a plurality of second data storage units in a target database, and primary key identifiers of corresponding primary keys of the first data storage units; based on the primary key identifier, acquiring total data corresponding to all total events occurring in the first data storage unit before the data migration starting time point; starting from the data migration starting time point, monitoring an incremental event occurring in the first data storage unit in real time, and obtaining incremental data corresponding to the incremental event; and migrating the total data and the incremental data into the target database to reproduce the total event and the incremental event in the second data storage unit. According to the method provided by the invention, the integrity of the data migrated to the target database can be ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data migration method, device and computer equipment. Background Art

[0002] MongoDB is a NoSQL (Not Only SQL, non-relational database management system) database, which is often used as the source database and target database in the database synchronization process. The existing database synchronization method is to migrate the existing data in the source database to the target database, but this method may result in incomplete data migrated to the target database. Summary of the invention

[0003] Based on this, it is necessary to provide a data migration method, device and computer equipment that can completely migrate data from a source database to a target database in response to the above technical problems.

[0004] In a first aspect, the present application provides a data migration method, the method comprising:

[0005] Acquire multiple first data storage units in the source database, multiple second data storage units in the target database, and primary key identifiers of corresponding primary keys of the first data storage units;

[0006] Based on the primary key identifier, obtaining full data corresponding to all full events occurring in the first data storage unit before the data migration start time point;

[0007] Starting from the start time point of the data migration, monitoring the incremental events occurring in the first data storage unit in real time, and acquiring the incremental data corresponding to the incremental events;

[0008] The full data and the incremental data are migrated to the target database to reproduce the full events and the incremental events in the second data storage unit.

[0009] In one embodiment, obtaining a plurality of second data storage units in the target database includes:

[0010] Acquire the table structure and index information of the first data storage unit;

[0011] The second data storage unit is generated in the target database based on the table structure and the index information.

[0012] In one of the embodiments, obtaining, based on the primary key identifier, full data corresponding to all full events occurring in the first data storage unit before the data migration start time point includes:

[0013] Based on the primary key identifier, obtaining a storage order of the first data storage unit in the source database;

[0014] Based on a maximum allocatable number of data storage units in a storage interval, dividing the plurality of first data storage units into a plurality of storage intervals according to the storage order;

[0015] For each storage interval, obtain the full amount of data corresponding to the full amount of events that occurred in the storage interval before the data migration start time point.

[0016] In one embodiment, the method further comprises:

[0017] For each storage interval, if all data corresponding to the storage interval cannot be completely migrated to the target database, obtaining a migration status corresponding to the storage interval and remaining data in the storage interval that has not been migrated to the target database;

[0018] Based on the migration status, all data corresponding to the storage interval is re-migrated to the target database, or the remaining data is migrated to the target database.

[0019] In one embodiment, obtaining the incremental data corresponding to the incremental event includes:

[0020] For multiple incremental events corresponding to the same primary key in the source database, the incremental data corresponding to the multiple incremental events are merged.

[0021] In one embodiment, for multiple incremental events corresponding to the same primary key in the source database, merging the incremental data corresponding to the multiple incremental events includes:

[0022] In the case where the incremental event occurs after the data insertion event, if the incremental event is a data update event, merging the incremental data corresponding to the data insertion event and the incremental data corresponding to the incremental event;

[0023] In the case where the incremental event occurs after a data insertion event or a data update event, if the incremental event is a data deletion event, based on the incremental data corresponding to the incremental event, the data insertion event or the data update event and the incremental event are merged into one data deletion event;

[0024] In the case where the incremental event occurs after the data update event, if the incremental event is a data update event, merging the incremental data corresponding to the data update event occurring before the incremental event and the incremental data corresponding to the incremental event;

[0025] In the case where the incremental event occurs after a data deletion event, if the incremental event is a data insertion event, the incremental data corresponding to the data deletion event and the incremental data corresponding to the incremental event are merged, and a data insertion event is generated based on the merged incremental data.

[0026] In one embodiment, the incremental event includes a data operation DML event and a data definition DDL event; and migrating the incremental data to the target database includes:

[0027] Obtain at least one existing DML event that has occurred before the current time point but has not been migrated to the target database;

[0028] When the number of the existing DML events is greater than a preset number, or the waiting time of the existing DML events is greater than a preset time, the incremental data corresponding to the at least one existing DML event is cached in a cache queue;

[0029] In the case where a DDL event occurs at the current time point, the incremental data corresponding to the at least one existing DML event and the incremental data corresponding to the currently occurring DDL event are cached in the cache queue;

[0030] The data cached in the cache queue is migrated to the target database.

[0031] In one embodiment, the cache queue includes a plurality of cache locations; the method further includes:

[0032] When there is no free cache position in the cache queue, the monitoring process of the first data storage unit is stopped until there is a free cache position in the cache queue.

[0033] In a second aspect, the present application further provides a data migration device, the device comprising:

[0034] A first acquisition module is used to acquire a plurality of first data storage units in a source database, a plurality of second data storage units in a target database, and primary key identifiers of corresponding primary keys of the first data storage units;

[0035] A second acquisition module is used to acquire, based on the primary key identifier, full data corresponding to all full events occurring in the first data storage unit before the start time point of data migration;

[0036] A monitoring module, configured to monitor, starting from the start time point of the data migration, the incremental events occurring in the first data storage unit in real time, and obtain the incremental data corresponding to the incremental events;

[0037] A data migration module is used to migrate the full data and the incremental data to the target database to reproduce the full events and the incremental events in the second data storage unit.

[0038] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0039] The above-mentioned data migration method, device and computer equipment obtain multiple first data storage units in the source database, multiple second data storage units in the target database, and primary key identifiers of the corresponding primary keys of the first data storage units; based on the primary key identifiers, obtain the full data corresponding to all full events that occurred in the first data storage unit before the start time point of data migration; starting from the start time point of data migration, monitor the incremental events that occurred in the first data storage unit in real time and obtain the incremental data corresponding to the incremental events; migrate the full data and incremental data to the target database to reproduce the full events and incremental events in the second data storage unit. The method provided by the present application can ensure the integrity of the data migrated to the target database. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of a data migration method in one embodiment;

[0042] Figure 2 A schematic diagram of a flow chart of a method for obtaining a second data storage unit in one embodiment;

[0043] Figure 3 is a flow chart of a data migration method in another embodiment;

[0044] Figure 4 is a structural block diagram of a data migration device in one embodiment;

[0045] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] In one embodiment, Figure 1 As shown, a data migration method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0048] S102: Acquire multiple first data storage units in the source database, multiple second data storage units in the target database, and primary key identifiers of corresponding primary keys of the first data storage units.

[0049] Optionally, in a MongoDB database, the data storage unit is a collection, and in a relational database, the data storage unit is a database table; a collection includes multiple documents, each document corresponds to a primary key, and the primary key is used to identify the corresponding document.

[0050] S104. Based on the primary key identifier, obtain full data corresponding to all full events that occurred in the first data storage unit before the data migration start time point.

[0051] Optionally, the full amount of data may be divided into multiple migration batches based on the primary key identifier, and the full amount of data may be migrated to the target database in batches.

[0052] S106 . Starting from the start time point of data migration, monitor incremental events occurring in the first data storage unit in real time, and obtain incremental data corresponding to the incremental events.

[0053] Optionally, the incremental events include DDL (Data Definition Language) events and DML (Data Manipulation Language) events. DDL events involve changes to the database schema, that is, changes to the database structure, and DML events involve changes to the data content in the database. Among them, DML events include data deletion events, data insertion events, and data update events; data deletion events include delete events and drop events. The delete event represents the deletion of part of the data in the data storage unit, and the drop event represents the deletion of all the data in the data storage unit.

[0054] S108. Migrate the full data and the incremental data to the target database to reproduce the full events and the incremental events in the second data storage unit.

[0055] Optionally, in the process of reproducing an incremental event, if the incremental event is a data insertion event or a data update event, an upsert operation is used to determine whether data corresponding to the incremental event exists in the target database. If so, an update operation is performed; if not, an insert operation is performed.

[0056] In the above data migration method, multiple first data storage units in the source database, multiple second data storage units in the target database, and primary key identifiers of the corresponding primary keys of the first data storage units are obtained; based on the primary key identifiers, the full data corresponding to all full events that occurred in the first data storage unit before the start time point of data migration are obtained; starting from the start time point of data migration, the incremental events that occurred in the first data storage unit are monitored in real time, and the incremental data corresponding to the incremental events are obtained; the full data and incremental data are migrated to the target database to reproduce the full events and incremental events in the second data storage unit. The method provided by the present application can ensure the integrity of the data migrated to the target database.

[0057] In some embodiments, Figure 2 As shown, obtaining multiple second data storage units in the target database includes:

[0058] S202: Obtain the table structure and index information of the first data storage unit.

[0059] S204: Generate a second data storage unit in the target database based on the table structure and index information.

[0060] Optionally, in a MongoDB database, the table structure refers to the structural information of a collection, and in a relational database, the table structure refers to the structural information of a database table; an index is used to improve query efficiency, for example, the index information may include an index field, an index type, and an index direction.

[0061] In this embodiment, based on the table structure and index information of the first data storage unit, a second data storage unit is generated in the target database so that the first data storage unit and the second data storage unit match, thereby improving the efficiency and accuracy of data migration.

[0062] In some embodiments, based on a primary key identifier, full data corresponding to all full events occurring in a first data storage unit before a start time point of data migration is obtained, including: based on the primary key identifier, obtaining a storage order of the first data storage unit in a source database; based on a maximum allocatable number of data storage units in a storage interval, dividing multiple first data storage units into multiple storage intervals in storage order; for each storage interval, obtaining full data corresponding to full events occurring in the storage interval before a start time point of data migration.

[0063] Optionally, multiple first data storage units are divided into multiple storage intervals, that is, the full data corresponding to the multiple first data storage units are divided into different storage intervals. During the data migration process, the full data is migrated to the target database in batches with the storage interval as the migration unit.

[0064] In this embodiment, the storage interval is used as the migration unit, and the full amount of data is migrated to the target database in batches, which can effectively avoid the problem of failure of full data migration due to excessive amount of data to be migrated.

[0065] In some embodiments, the method also includes: for each storage interval, when the full amount of data corresponding to the storage interval cannot be completely migrated to the target database, obtaining the migration status corresponding to the storage interval and the remaining data in the storage interval that has not been migrated to the target database; based on the migration status, re-migrating the full amount of data corresponding to the storage interval to the target database, or migrating the remaining data to the target database.

[0066] Optionally, if all the data corresponding to the storage interval cannot be migrated to the target database, it may be caused by the user actively suspending the data migration process or by external reasons. Depending on the reason, it can be determined whether all the data should be re-migrated or only the remaining data that has not been migrated should be migrated.

[0067] In this embodiment, based on the migration status, the full amount of data corresponding to the storage interval is re-migrated to the target database, or the remaining data is migrated to the target database, which can ensure that all the full amount of data is migrated to the target database and improve the migration efficiency.

[0068] In some embodiments, obtaining incremental data corresponding to an incremental event includes: for multiple incremental events corresponding to the same primary key in a source database, merging the incremental data corresponding to the multiple incremental events.

[0069] Optionally, the purpose of merging the incremental data of multiple incremental events corresponding to the same primary key in the source database is to merge multiple incremental events corresponding to the same primary key, so that when the target database reproduces the event, it will not execute each incremental event one by one, but directly execute the merged incremental event, which can effectively reduce the number of incremental events executed by the target database and improve execution efficiency.

[0070] In this embodiment, for multiple incremental events corresponding to the same primary key in the source database, the incremental data corresponding to the multiple incremental events are merged, which can effectively reduce the number of incremental events executed by the target database and improve execution efficiency.

[0071] In some embodiments, for multiple incremental events corresponding to the same primary key in the source database, the incremental data corresponding to the multiple incremental events are merged, including: when the incremental event occurs after a data insertion event, if the incremental event is a data update event, the incremental data corresponding to the data insertion event and the incremental data corresponding to the incremental event are merged; when the incremental event occurs after a data insertion event or a data update event, if the incremental event is a data deletion event, based on the incremental data corresponding to the incremental event, the data insertion event or the data update event and the incremental event are merged into one data deletion event; when the incremental event occurs after a data update event, if the incremental event is a data update event, the incremental data corresponding to the data update event occurring before the incremental event and the incremental data corresponding to the incremental event are merged; when the incremental event occurs after a data deletion event, if the incremental event is a data insertion event, the incremental data corresponding to the data deletion event and the incremental data corresponding to the incremental event are merged, and one data insertion event is generated based on the merged incremental data.

[0072] Optionally, the incremental data corresponding to different incremental events are merged according to the data merging rules in this embodiment. For each merged incremental data, the merged incremental data is synchronized to the target database as a migration batch, and the merged incremental data is reproduced in the target database as data corresponding to a database event.

[0073] In this embodiment, the merged incremental data is synchronized to the target database as a migration batch, which can effectively avoid the problem of incremental data migration failure due to excessive amount of migrated data; the merged incremental data is reproduced in the target database as the data corresponding to a database event, which can save computing resources.

[0074] In some embodiments, incremental events include data operation DML events and data definition DDL events; migrating incremental data to a target database includes: obtaining at least one existing DML event that has occurred before the current time point but has not been migrated to the target database; when the number of existing DML events is greater than a preset number, or the waiting time of an existing DML event is greater than a preset time, caching the incremental data corresponding to at least one existing DML event into a cache queue; when a DDL event occurs at the current time point, caching both the incremental data corresponding to at least one existing DML event and the incremental data corresponding to the currently occurring DDL event into the cache queue; and migrating the data cached in the cache queue to the target database.

[0075] Optionally, the waiting time refers to the time interval between the occurrence time point of the existing DML event and the current time point; the cache queue is a temporary storage structure used to cache data read during the migration process and data to be written to the target database. For example, the cache queue can be a ring cache queue.

[0076] In this embodiment, after the incremental data is read and before the incremental data is migrated to the target database, the incremental data is temporarily stored in the cache queue according to the rules in this embodiment, which can effectively improve the reliability of data migration.

[0077] In some embodiments, the cache queue includes multiple cache locations; the method further includes: when there is no free cache location in the cache queue, stopping the monitoring process of the first data storage unit until there is a free cache location in the cache queue.

[0078] Optionally, if there is no free cache position in the cache queue, it means that the cache queue is full. In this case, the monitoring process needs to be suspended. When the cache data in the cache position is written into the target database, the monitoring process for the source database is re-awakened.

[0079] In this embodiment, when there is no free cache position in the cache queue, the monitoring process of the first data storage unit is stopped, which can effectively improve the reliability of data migration.

[0080] In one embodiment, Figure 3 As shown, another data migration method is provided, which includes the following:

[0081] (1) A table structure migration subsystem: reads the table structure and index information of the source MongoDb database and replays and rebuilds it in the target database. The process is to traverse all source collections, read the collection index information, create a collection with the same name in the target database, and create an index in the target database collection.

[0082] (2) A full-data synchronization subsystem with breakpoint-resume function: By utilizing the characteristics of the MongoDB primary key _id, the Collection is segmented and fully synchronized, thus achieving full-data synchronization that supports parallelization and breakpoint-resume function.

[0083] 1) Traverse the Collection according to the set step size to determine the start and end _id of each interval.

[0084] 2) Execute the full task for each interval, and different intervals can be executed in parallel; after the task is completed, the interval completion status is recorded.

[0085] 3) When the user pauses or external reasons cause full synchronization to be interrupted, the breakpoint can be resumed according to the interval task status.

[0086] (3) An incremental synchronization subsystem that supports DDL and DML

[0087] 1) Define a set of stream event listeners and extractors: listen to, group, and parse DDL and DML events, and extract resources and control traffic through a circular cache queue.

[0088] DDL increment: When subscribing to Change Stream, turn on the ShowExpandedEvents property to subscribe to DDL events. When incremental extraction occurs, a single batch is processed when a DDL event is encountered.

[0089] DML incremental: Subscribe to and listen to DML events in the Change Stream, including delete, drop, insert, update, etc. For insert and update events, use the Upsert operation in the target MongoDB. The Upsert operation makes multiple incremental synchronization operations idempotent, thereby supporting site adjustments without affecting the execution of incremental synchronization.

[0090] 2) Define a set of efficient idempotent loaders: process and group incremental data according to Collection and _id, and use Replace and Upsert operations to store the processed and grouped data, which improves the efficiency and success rate of incremental storage and can perform idempotent operations on the same incremental data. The storage method is:

[0091] a) Incremental data in the same batch is grouped by Collection Name.

[0092] b) For the groups in a), perform the merge operation according to the primary key _Id. The merge algorithm is:

[0093] i. If an insert event is followed by an update or replace event, they are combined into an insert operation, and the insert data is the full Document information of the second event.

[0094] ii. If a delete event occurs after an insert or update event, they are merged into a delete event.

[0095] iii. Multiple update or replace events are merged into one update event, and the data is the full Document information of the last update event.

[0096] iv. A delete event and an insert event are merged into an insert event.

[0097] c) Further group the data after merging in b).

[0098] The final groups are stored in parallel.

[0099] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0100] Based on the same inventive concept, the embodiment of the present application also provides a data migration device for implementing the data migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data migration device embodiments provided below can refer to the limitations on the data migration method above, and will not be repeated here.

[0101] In an exemplary embodiment, Figure 4 As shown, a data migration device 400 is provided, comprising: a first acquisition module 401, a second acquisition module 402, a monitoring module 403 and a data migration module 404, wherein:

[0102] The first acquisition module 401 is used to acquire a plurality of first data storage units in a source database, a plurality of second data storage units in a target database, and primary key identifiers of primary keys corresponding to the first data storage units.

[0103] The second acquisition module 402 is used to acquire, based on the primary key identifier, full data corresponding to all full events occurring in the first data storage unit before the data migration start time point.

[0104] The monitoring module 403 is used to monitor the incremental events occurring in the first data storage unit in real time starting from the start time point of the data migration, and obtain the incremental data corresponding to the incremental events.

[0105] The data migration module 404 is used to migrate the full data and the incremental data to the target database to reproduce the full events and the incremental events in the second data storage unit.

[0106] In some embodiments, the first acquisition module 401 is further used to acquire the table structure and index information of the first data storage unit; and generate the second data storage unit in the target database based on the table structure and the index information.

[0107] In some embodiments, the second acquisition module 402 is also used to obtain the storage order of the first data storage unit in the source database based on the primary key identifier; based on the maximum allocatable number of data storage units in the storage interval, divide the multiple first data storage units into multiple storage intervals according to the storage order; for each storage interval, obtain the full data corresponding to the full events that occurred in the storage interval before the start time point of the data migration.

[0108] In some embodiments, the second acquisition module 402 is also used to obtain, for each storage interval, a migration status corresponding to the storage interval and remaining data in the storage interval that has not been migrated to the target database when the full amount of data corresponding to the storage interval has not been completely migrated to the target database; based on the migration status, re-migrate the full amount of data corresponding to the storage interval to the target database, or migrate the remaining data to the target database.

[0109] In some embodiments, the monitoring module 403 includes:

[0110] The merging unit is used to merge the incremental data corresponding to the multiple incremental events corresponding to the same primary key in the source database.

[0111] In some embodiments, the merging unit is further used for, in the case where the incremental event occurs after the data insertion event, if the incremental event is a data update event, merging the incremental data corresponding to the data insertion event with the incremental data corresponding to the incremental event; in the case where the incremental event occurs after the data insertion event or the data update event, if the incremental event is a data deletion event, merging the data insertion event or the data update event with the incremental event into a data deletion event based on the incremental data corresponding to the incremental event; in the case where the incremental event occurs after the data update event, if the incremental event is a data update event, merging the incremental data corresponding to the data update event that occurred before the incremental event with the incremental data corresponding to the incremental event; in the case where the incremental event occurs after the data deletion event, if the incremental event is a data insertion event, merging the incremental data corresponding to the data deletion event with the incremental data corresponding to the incremental event, and generating a data insertion event based on the merged incremental data.

[0112] In some embodiments, the incremental events include data operation DML events and data definition DDL events; the data migration module 404 is also used to obtain at least one existing DML event that has occurred before the current time point but has not been migrated to the target database; when the number of the existing DML events is greater than a preset number, or the waiting time of the existing DML events is greater than a preset time, the incremental data corresponding to the at least one existing DML event is cached in a cache queue; when a DDL event occurs at the current time point, the incremental data corresponding to the at least one existing DML event and the incremental data corresponding to the currently occurring DDL event are both cached in the cache queue; and the data cached in the cache queue is migrated to the target database.

[0113] In some embodiments, the cache queue includes multiple cache locations; the data migration module 404 is also used to stop the monitoring process of the first data storage unit when there is no free cache location in the cache queue until there is a free cache location in the cache queue.

[0114] Each module in the above data migration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0115] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a data migration method is implemented.

[0116] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0119] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0121] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data migration method, characterized in that: The method comprises: Acquire multiple first data storage units in the source database, multiple second data storage units in the target database, and primary key identifiers of corresponding primary keys of the first data storage units; Based on the primary key identifier, obtaining full data corresponding to all full events occurring in the first data storage unit before the data migration start time point; Starting from the start time point of the data migration, monitoring the incremental events occurring in the first data storage unit in real time, and acquiring the incremental data corresponding to the incremental events; The full data and the incremental data are migrated to the target database to reproduce the full events and the incremental events in the second data storage unit.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of second data storage units in the target database comprises: Acquire the table structure and index information of the first data storage unit; The second data storage unit is generated in the target database based on the table structure and the index information.

3. The method according to claim 1, characterized in that The acquiring, based on the primary key identifier, full data corresponding to all full events occurring in the first data storage unit before the data migration start time point includes: Based on the primary key identifier, obtaining a storage order of the first data storage unit in the source database; Based on a maximum allocatable number of data storage units in a storage interval, dividing the plurality of first data storage units into a plurality of storage intervals according to the storage order; For each storage interval, obtain the full amount of data corresponding to the full amount of events that occurred in the storage interval before the data migration start time point.

4. The method according to claim 3, characterized in that The method further comprises: For each storage interval, if all data corresponding to the storage interval cannot be completely migrated to the target database, obtaining a migration status corresponding to the storage interval and remaining data in the storage interval that has not been migrated to the target database; Based on the migration status, all data corresponding to the storage interval is re-migrated to the target database, or the remaining data is migrated to the target database.

5. The method according to claim 1, characterized in that The obtaining of the incremental data corresponding to the incremental event includes: For multiple incremental events corresponding to the same primary key in the source database, the incremental data corresponding to the multiple incremental events are merged.

6. The method according to claim 5, characterized in that The merging of the incremental data corresponding to the multiple incremental events corresponding to the same primary key in the source database comprises: In the case where the incremental event occurs after the data insertion event, if the incremental event is a data update event, merging the incremental data corresponding to the data insertion event and the incremental data corresponding to the incremental event; In the case where the incremental event occurs after a data insertion event or a data update event, if the incremental event is a data deletion event, based on the incremental data corresponding to the incremental event, the data insertion event or the data update event and the incremental event are merged into one data deletion event; In the case where the incremental event occurs after the data update event, if the incremental event is a data update event, merging the incremental data corresponding to the data update event occurring before the incremental event and the incremental data corresponding to the incremental event; In the case where the incremental event occurs after a data deletion event, if the incremental event is a data insertion event, the incremental data corresponding to the data deletion event and the incremental data corresponding to the incremental event are merged, and a data insertion event is generated based on the merged incremental data.

7. The method according to claim 1, characterized in that The incremental events include data operation DML events and data definition DDL events; and migrating the incremental data to the target database includes: Obtain at least one existing DML event that has occurred before the current time point but has not been migrated to the target database; When the number of the existing DML events is greater than a preset number, or the waiting time of the existing DML events is greater than a preset time, the incremental data corresponding to the at least one existing DML event is cached in a cache queue; In the case where a DDL event occurs at the current time point, the incremental data corresponding to the at least one existing DML event and the incremental data corresponding to the currently occurring DDL event are cached in the cache queue; The data cached in the cache queue is migrated to the target database.

8. The method according to claim 7, characterized in that The cache queue includes a plurality of cache locations; the method further includes: When there is no free cache position in the cache queue, the monitoring process of the first data storage unit is stopped until there is a free cache position in the cache queue.

9. A data migration device, characterized in that: The device comprises: A first acquisition module is used to acquire a plurality of first data storage units in a source database, a plurality of second data storage units in a target database, and primary key identifiers of corresponding primary keys of the first data storage units; A second acquisition module is used to acquire, based on the primary key identifier, full data corresponding to all full events occurring in the first data storage unit before the start time point of data migration; A monitoring module, configured to monitor, starting from the start time point of the data migration, the incremental events occurring in the first data storage unit in real time, and obtain the incremental data corresponding to the incremental events; A data migration module is used to migrate the full data and the incremental data to the target database to reproduce the full events and the incremental events in the second data storage unit.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images