Data migration method and related device

Abstract

The invention discloses a data migration method and a related device, and relates to the technical field of data processing, stock data in a source database before a first time point is migrated to a target database, increment operation on the source database after the first time point is recorded to an increment table by adopting a database interceptor, and the increment table is decoupled from business logic, so that the data migration efficiency is improved. According to the method, online service demand codes are not affected, after stock data migration is finished, incremental data corresponding to incremental operation in the incremental table is migrated to the target database, and it is ensured that incremental data generated in the migration process is completely migrated. And determining a second time point according to the migration progress of the incremental data, forwarding the flow accessed to the source database to the target database at the second time point, and realizing data migration under the condition that the system is not shut down through configuration of two parts of data migration and flow switching.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data migration method and related apparatus. Background Technology

[0002] With the development of information technology, the amount of data accumulated by business systems is growing larger and larger, and centralized databases can hardly meet the system requirements, resulting in a need for transformation from centralized to distributed architecture.

[0003] In the process of database upgrade and transformation, data migration is a core and critical step, which is usually accompanied by changes in data structure. Traditional data migration methods require data to be static when handling database data migration and processing, resulting in application downtime and affecting the normal operation of business systems. Summary of the Invention

[0004] In view of the above problems, this application provides a data migration method and related apparatus to achieve data migration without system shutdown. The specific solution is as follows:

[0005] The first aspect of this application provides a data migration method, including:

[0006] Migrate existing data in the source database prior to the first time point to the target database;

[0007] A database interceptor is used to record incremental operations on the source database after the first time point to an incremental table;

[0008] After all the existing data has been migrated to the target database, the incremental data corresponding to the incremental operations in the incremental table will be migrated to the target database.

[0009] The second time point is determined based on the migration progress of the incremental data;

[0010] At the second time point, traffic accessing the source database is forwarded to the target database.

[0011] In one possible implementation, migrating existing data from the source database to the target database prior to the first time point includes:

[0012] The existing data to be migrated is obtained based on the order of the primary keys in the source database;

[0013] Based on the sharding rules of the data table corresponding to the existing data to be migrated, determine the physical sharding of the existing data to be migrated in the target database;

[0014] The batch submission method is used to call each physical shard processing thread to write the existing data to be migrated into the corresponding physical shard.

[0015] In one possible implementation, migrating existing data from the original database prior to the first time point to the target database further includes:

[0016] If an abnormal exit is detected, obtain the data migration status;

[0017] Obtain the breakpoint data location information based on the data migration status;

[0018] The step of obtaining the existing data to be migrated in sequence based on the data primary key in the source database is triggered according to the breakpoint data location information, until all the existing data is migrated to the target database.

[0019] In one possible implementation, migrating the incremental data corresponding to the incremental operation in the incremental table to the target database includes:

[0020] Retrieve the incremental operation data recorded in the incremental table;

[0021] The incremental operation data is parsed to obtain the location conditions of the incremental operation data in the source database;

[0022] Obtain the incremental data to be migrated based on the positioning conditions;

[0023] The incremental data to be migrated is migrated to the target database.

[0024] In one possible implementation, determining the second time point based on the migration progress of the incremental data includes:

[0025] If there is less than one batch of incremental data remaining to be migrated, the current time point will be designated as the second time point.

[0026] In one possible implementation, forwarding the traffic accessed by the source database to the target database at the second time point includes:

[0027] At the second time point, the source database's filter is used to forward traffic accessing the source database to the target database;

[0028] Use a database interceptor to intercept write operations to the target database;

[0029] If all incremental data corresponding to all incremental operations in the incremental table has been migrated to the target database, then a write operation is performed on the target database.

[0030] If not all incremental data corresponding to all incremental operations in the incremental table has been migrated to the target database, then write operations to the target database will be refused.

[0031] A second aspect of this application provides a data migration apparatus, comprising:

[0032] The first migration unit is used to migrate the existing data in the source database before the first time point to the target database.

[0033] An interception unit is used to record incremental operations on the source database after the first time point into an incremental table using a database interceptor.

[0034] The second migration unit is used to migrate the incremental data corresponding to the incremental operation in the incremental table to the target database after all the existing data has been migrated to the target database.

[0035] The determining unit is used to determine a second time point based on the migration progress of the incremental data;

[0036] The flow-cutting unit is used to forward traffic accessing the source database to the target database at the second time point.

[0037] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the data migration method described in the first aspect or any implementation thereof.

[0038] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the data migration method of the first aspect or any implementation thereof.

[0041] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the data migration method described in the first aspect or any implementation thereof.

[0042] Using the above technical solution, this application provides a data migration method and related apparatus that migrates existing data in the source database before a first time point to the target database. By employing a database interceptor, incremental operations on the source database after the first time point are recorded in an incremental table, decoupled from business logic and without affecting online business requirement code. After the existing data migration is complete, the incremental data corresponding to the incremental operations in the incremental table is migrated to the target database, ensuring that incremental data generated during the migration process is also completely migrated. A second time point is determined based on the migration progress of the incremental data. At the second time point, traffic accessing the source database is forwarded to the target database. Through the configuration of both data migration and process switching, data migration without system downtime is achieved. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A flowchart illustrating a data migration method provided in an embodiment of this application;

[0045] Figure 2 This is a schematic diagram of a data migration scenario provided in an embodiment of this application;

[0046] Figure 3 This application provides a schematic diagram of a data migration process as an embodiment of the present application.

[0047] Figure 4 This is a schematic diagram of the laying-up process provided in the embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the number pursuit process provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the cloud synchronization strategy provided in an embodiment of this application. Detailed Implementation

[0050] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0051] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0052] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0053] In practical applications, if a customer service system has high requirements for both business continuity and data availability—such as a 24 / 7 customer service system—its business continuity requirements are high, and external demands cannot be frozen. Therefore, it's crucial to avoid application downtime leading to service unavailability during data migration. Several non-downtime data migration methods exist, but all have some limitations.

[0054] Technical Solution 1: The old credit card system determines the data tables to be migrated based on the customer whitelist, converts them into migration files, and saves them in batches to an intermediate network storage. Then, the new credit card business system reads the migration files from the network storage, transcodes them, and saves them to the new data tables.

[0055] Technical Solution 2: A file data migration system was built between the source and target NAS system storage, realizing smooth, non-stop file data migration.

[0056] Technical Solution 3: Store the transaction business attribute data from the original system in a data storage warehouse, then obtain the offline table data from the data storage warehouse and migrate it to the data system to which the data to be migrated will be located.

[0057] Technical Solution 4: Migrate all data to an intermediate database, deploy Canal middleware to listen to the MySQL Binlog generated by the intermediate database, and store it in a message queue. Then, develop a custom migration program to parse the Binlog in the message queue, convert it into the required data structure, and store it in the target database.

[0058] Both technical solutions one and two rely on external storage devices, increasing data migration costs. Furthermore, they place high demands on the stability of the external storage media during implementation; data loss is risked if the external facilities experience power outages or damage. Additionally, solution two primarily addresses the migration of file data, not database data, making it unsuitable for projects requiring database replacement.

[0059] Technical Solution 3 relies on a data storage warehouse during implementation. If the organization does not already have a data warehouse, it needs to be built from scratch, increasing the workload during data migration. Furthermore, this solution primarily acquires offline data from the data storage warehouse during the migration to the target database, and cannot obtain near real-time data from the target database, resulting in data latency.

[0060] Technical Solution 4 requires the introduction of an intermediate MySQL database during implementation. If neither the source nor the target database is a MySQL database, a new database type will be introduced, increasing the risk of data migration. This solution also requires the deployment of DTS (Data Transformation Services) and the introduction of Canal and message queue middleware. Throughout the migration process, there are numerous external dependencies, demanding a high level of experience and expertise from developers, and making the project prone to technical risks.

[0061] In summary, there is currently no lightweight, non-disruptive data migration solution that supports heterogeneous databases and heterogeneous data structures.

[0062] To address the aforementioned problems, embodiments of this application provide a data migration method. The data migration method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0063] Reference Figure 1 , Figure 1 This is a flowchart illustrating a data migration method provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiment of this application, a data migration method may include steps 101 to 105, which are described in detail below.

[0064] 101: Migrate existing data in the source database prior to the first time point to the target database;

[0065] There are two key time points in the data migration process: the time when the data migration begins (i.e., the first time point) and the time when the traffic is switched (i.e., the second time point).

[0066] Let T1 represent the first time point. Then the data written in the source database at the first time point is the existing data D(T1).

[0067] The source and target databases can be heterogeneous. For example, the source database can be an Oracle database with a monolithic architecture in the cloud, while the target database can be a TDSQL distributed database.

[0068] Specifically, by executing the data migration procedure, the existing data in the source database before the first time point is migrated to the target database in an independent batch manner, thus decoupling it from the system's online transactions.

[0069] 102: Use a database interceptor to record incremental operations on the source database after the first time point to the incremental table;

[0070] Let T2 represent the second time point, then the data generated by the source database between T1 and T2 is the incremental data Δ(T).

[0071] That is, the total amount of data to be migrated is D(All) = D(T1) + Δ(T).

[0072] 103: After all existing data has been migrated to the target database, migrate the incremental data corresponding to the incremental operations in the incremental table to the target database.

[0073] The migration of incremental data also adopts an independent batch approach, specifically implemented using a tracking program.

[0074] 104: Determine the second time point based on the migration progress of incremental data;

[0075] The key to the traffic switching process is the selection of the switching time point T2. Optimal T2 should be the point when all incremental data has been migrated to the target database. However, in a production environment, business operations continue uninterrupted, and incremental data is continuously generated. The optimal switching time point T2 would need to be the exact moment the data migration is complete, with no users making transactions—an unpredictable and unrealistic scenario. Therefore, this embodiment selects a point when the tracking process is essentially complete as the switching time point T2 based on the incremental data migration progress: when the last batch of data in the incremental table is less than a complete batch, the system traffic is switched. If the switching time point is earlier than this, a significant amount of data in the source database has not yet been migrated to the target database, causing the target database to remain unwritable for an extended period, potentially leading to a decrease in transaction success rate. If the switching time point is later than this, the source database will continue to generate new incremental data, requiring the tracking process to continuously synchronize the database, prolonging the system's migration time and increasing deployment risk. Therefore, the selected traffic switching time point in this embodiment can effectively balance the two extreme cases mentioned above, and complete the traffic switching under controllable risk.

[0076] 105: At the second time point, forward the traffic accessing the source database to the target database.

[0077] Traditional traffic switching solutions struggle to perform a clean and instantaneous switch of all traffic without system downtime, potentially leading to data inconsistency risks. Therefore, this embodiment employs a combination of "original system traffic forwarding + consumer address switching," completing the system traffic switching task in two parts to achieve a seamless switching experience for users and avoid the risks associated with intermediate data transmission.

[0078] This embodiment provides a data migration method that migrates existing data in the source database before a first time point to the target database. By employing a database interceptor, incremental operations on the source database after the first time point are recorded in an incremental table, decoupling them from business logic and avoiding impact on online business requirement code. After the existing data migration is complete, the incremental data corresponding to the incremental operations in the incremental table is migrated to the target database, ensuring that incremental data generated during the migration process is also completely migrated. A second time point is determined based on the incremental data migration progress. At this second time point, traffic accessing the source database is forwarded to the target database. Through the configuration of both data migration and process switching, data migration can be achieved without introducing intermediate databases or data warehouses, while ensuring system continuity.

[0079] The following example, using an Oracle database as the source database, a monolithic on-premises architecture, and a 24 / 7 customer service system as an example, details the data migration method provided in the above embodiment. The method involves migrating data from the source database to a TDSQL distributed database and upgrading and reconstructing the original system using a distributed cloud architecture. For ease of explanation, the source database is referred to as the on-premises database, corresponding to the on-premises system, and the target database is referred to as the cloud-based database, corresponding to the cloud-based system.

[0080] Please see Figure 2 The diagram illustrates a data migration scenario, where S1-S4 represent four physical shards of the cloud-based distributed database (i.e., the target database). In this embodiment, data migration includes data migration and traffic switching. Before traffic switching, the original system data (i.e., the data in the source database) must first be migrated. Figure 2 Step ①) Migrate all existing data at the current point in time, while ensuring that incremental business data generated during the migration process is also completely migrated, so as to ensure that the data is not duplicated, omitted, or disordered.

[0081] After data migration is complete, traditional traffic switching solutions cannot guarantee that all consumer-related parties will switch traffic simultaneously. To avoid data consistency issues, all traffic should be synchronously and completely switched to the cloud after data migration. The system needs to build its own traffic switching component. After data migration is complete, traffic from the on-premises cloud will be sent to the cloud. Figure 2 Step ②), and then each related party switches traffic ( Figure 2Step ③). Through the combined efforts of data migration and traffic switching, the ultimate goal is to achieve a seamless system upgrade process for users without system downtime.

[0082] In summary, the data migration process includes three main steps: data migration, traffic forwarding, and traffic switching.

[0083] Please see Figure 3 The data migration flowchart shown below refers to the existing data D(T1) before time T1. After the data migration begins, incremental data Δ(T) is generated in the on-premises system until the traffic switching point T2. After time T2, since the traffic has been switched to the cloud system, all data will be written to the cloud system, and there will be no more incremental data on the on-premises system. During the migration of existing data D(T1), the cloud database is empty, so D(T1) is considered as laying the foundation for the new cloud database. Therefore, the migration of existing data D(T1) is simply referred to as laying the foundation. During the migration of incremental data Δ(T), the incremental data is used to track the dynamic changes of the foundation data. Therefore, the migration of incremental data is simply referred to as tracking the changes. For ease of explanation, the terms "laying the foundation" and "tracking the changes" will be used consistently below.

[0084] Please see Figure 4 The diagram shown illustrates the data distribution process, which includes the following steps: breakpoint resolution, sequential data acquisition, data distribution, data storage, and breakpoint reporting.

[0085] (1) Breakpoint reading

[0086] When dealing with large amounts of existing data, the data transfer process can be lengthy. To prevent unexpected issues and enhance the robustness of the data transfer program, a breakpoint resume function is added. If the data transfer program exits abnormally, it can resume from the breakpoint position the next time it is restarted.

[0087] When the number distribution program is invoked, it first needs to read the data position of the last breakpoint. Figure 4 Step ①). When the data averaging program exits abnormally, the primary key information of the breakpoint data is stored in the database. A custom splitting method is used in the program to concatenate the primary key values ​​of the breakpoint data using colons. The primary key values ​​of the breakpoint data are split again when the breakpoint data position is read next time.

[0088] It should be noted that when the data averaging program is started for the first time, the breakpoint data position is set to a predefined initial value, also known as the "zero point". The data averaging program then starts to obtain data sequentially from the first data at the "zero point".

[0089] (2) Sequential data acquisition

[0090] Data migration involves moving existing data, aiming to ensure no duplication, omissions, or data corruption during the process. In a database, each table has a corresponding index created for its primary key upon creation, ensuring the order of data stored. Furthermore, modifications to existing data during migration are recorded and processed in the data tracking program. Therefore, when migrating existing data, only the current point in time needs to be considered; incremental changes during data processing are unnecessary, and the primary key-based sorting remains reliable.

[0091] To achieve the target number of entries, this embodiment uses the primary key and follows the leftmost principle to retrieve and sequentially arrange the data. The existing data is then retrieved in batches according to this order to complete the migration of the existing data. The specific order rule based on the primary key is as follows: assuming the primary key in the data table is (a, b, c), and the breakpoint is (α, β, γ), then the existing data after that breakpoint is:

[0092] select*from table where(a=αand b=βand c>γ)or(a=αand b>β)or(a>α)order by a,b,c;

[0093] The initial value of (α,β,γ), also known as the zero-point value, will take different values ​​depending on the type of the primary key (a,b,c), as shown below:

[0094]

[0095] (3) Data distribution

[0096] Since the distributed database used in the cloud has four shards, it is necessary to consider which shard in the cloud corresponds to the on-premises data during the data migration process.

[0097] This embodiment determines the physical shards of the existing data to be migrated in the cloud based on the sharding rules of the data table corresponding to the existing data to be migrated. It makes full use of the database's batch commit capability, avoids cross-shard distributed transactions, greatly improves the overall efficiency of data migration, and reduces the duration of data migration. Figure 4 Step ③).

[0098] Specifically, after obtaining the existing data to be migrated, it's necessary to determine which data table the data belongs to. Then, based on the table's sharding rules, the shard key of the current data is calculated. Simultaneously, if the data structure of this table has changed during system upgrades, the data to be inserted into the cloud should also be transformed to adapt to the cloud's data table format. Finally, according to the mapping relationship between logical and physical shards in the distributed database, the existing data to be migrated, with its physical shard determined, is sent to the corresponding physical shard processing thread.

[0099] (4) Data storage

[0100] After determining the physical shards and sending the existing data to be migrated to the corresponding physical shard processing thread, the data is merged and submitted in batches using a batch submission method. Taking a cloud physical shard count of 4 as an example, 4 physical shard processing threads are used to submit data in batches of 1000 records each. Figure 4 Step 4). Compared to single-loop data writes, batch submission can significantly improve data writing efficiency.

[0101] (5) Breakpoint reporting

[0102] If an abnormal exit is detected, the data migration status is obtained, the breakpoint data location information is retrieved based on the data migration status, and submitted to the batch executor to facilitate the continued execution of the subsequent data spreading program. Figure 4 Step ⑤).

[0103] There are two main situations regarding the determination of breakpoint data locations:

[0104] If all the distributed data shards are successfully executed, the last data shard in the blocking queue will be reported as the breakpoint data position.

[0105] If data fails to be written to the database, use binary search to check the data that could not be written. Since the data is sorted according to a specific order during the writing process, using binary search to try writing in batches can identify the data that failed to be written at the beginning of the sorted list and submit the breakpoint data location.

[0106] In summary, the five steps of the data acquisition process—breakpoint reading, sequential data retrieval, data distribution, data storage, and breakpoint reporting—ensure the integrity of the existing data before the start of the data acquisition process.

[0107] Data migration refers to the migration of incremental data Δ(T), which mainly includes the migration of data generated by three types of operations—add, delete, and modify—in the on-premises database during the data migration process.

[0108] In one possible implementation, before the data migration begins, specifically at time point T1, the incremental operation recording switch for the on-premises system is activated. The database interceptor writes all incremental operations into the on-premises incremental table, which records the SQL statements for these incremental operations. After the data migration process is complete, i.e., after all existing data has been migrated to the cloud database, the follow-up data process begins, consuming the incremental operation SQL statements from the incremental table and migrating the relevant incremental data to the cloud database.

[0109] When consuming incremental data, if direct SQL replay is used for data migration, it's crucial to ensure that the incremental operation SQL data is written sequentially to the incremental table. Furthermore, during incremental data consumption, a single thread must be used to execute the SQL statements in the cloud database according to the order of the incremental operation tables. Adopting this approach requires adding control over the high-concurrency data write-to-database order, necessitating rate limiting and other measures for normal business transactions, which is costly. Moreover, using only a single thread during data tracing becomes the performance bottleneck of the tracing process.

[0110] To address the aforementioned issues and significantly improve the efficiency of data tracking, this embodiment employs a multi-threaded approach. It queries the business data corresponding to the incremental operation SQL in the on-premises database, inserts or updates the business data in the cloud database, and uses eventual consistency to achieve data synchronization, thereby enhancing tracking efficiency. Specifically, it retrieves the incremental operation data recorded in the incremental table, parses the incremental operation data to obtain its location conditions in the on-premises database, and then uses these location conditions to obtain the incremental data to be migrated. This data is then migrated to the cloud database, fully leveraging the capabilities of multi-threading to accelerate tracking efficiency while ensuring the correctness of incremental data migration.

[0111] In summary, please refer to the above. Figure 5 The flowchart shown above illustrates the data migration process. The data migration procedure includes five steps: incremental operation data storage, incremental operation data acquisition, incremental operation data parsing, incremental source data acquisition, and incremental data migration. This process migrates incremental data and ultimately completes the migration of all data.

[0112] (1) Incremental operation to the database

[0113] During the data rollout process, newly generated incremental data needs to be recorded to facilitate the migration of incremental data after the rollout is complete. Figure 5 Step ①). Specifically, a database interceptor is used to intercept all incremental operations on the on-premises database and persist them to the incremental table in the database. The structure of the incremental table is shown in Table 1.

[0114] Table 1. Table Structure of the Incremental Tracking Table

[0115]

[0116] To facilitate subsequent monitoring of incremental data synchronization, a data status column was added to the incremental table, marked using the `consume_status` property. To ensure that all incremental operations on business data are persisted, transactions are required for control.

[0117] (2) Incremental operation data acquisition

[0118] Retrieving incremental data from the incremental table is similar to retrieving data during the number expansion process. A blocking queue is used to improve the concurrency of retrieving incremental data and parsing subsequent data operations, thereby enhancing the overall efficiency of the program. Figure 5 Step ②).

[0119] (3) Incremental operation data parsing

[0120] This operation mainly involves parsing the operation data obtained from the incremental table. Figure 5 Step ③). Based on the information recorded in the incremental table, the incremental operations are first divided into three categories: insert, update, and delete. For different incremental operation data, the corresponding table name and WHERE condition in the incremental operation SQL are identified, i.e., the location condition. Then, based on the identified information, the corresponding SELECT query statement is constructed to facilitate subsequent querying of the source data.

[0121] (4) Source data acquisition

[0122] Execute the cloud-based database query statement obtained after parsing and assembling in the previous step to retrieve data from the cloud-based data table under the current location conditions. This data is the final value at the current moment. Figure 5 Step ④). Here, to improve the speed of source data acquisition, a thread pool + connection pool solution is adopted. During the data retrieval process, it is necessary not only to ensure the accuracy of the data but also to ensure that no data is missed. For each data retrieval task, it is necessary to ensure its successful execution. Therefore, the rejection policy of the thread pool needs to be set to: when the thread pool queue is full, newly added tasks need to be executed by the calling thread.

[0123] (5) Incremental data migration

[0124] The process of adding incremental data is similar to adding existing data during the data rollout process. First, the data table to which the current data belongs needs to be identified. Then, based on the sharding rules specified for that table, the shard to which the data belongs in the cloud is calculated, and data from different shards is packaged together. It should also be noted whether there have been any changes to the data table structure in the cloud, requiring a transformation of the cloud data structure. Finally, the packaged data is sent to its corresponding physical shard execution thread to reduce performance fluctuations in the data rollout process. Figure 5 Step ⑤).

[0125] In one possible implementation, the positioning condition obtained by parsing the incremental operation data in the incremental table is used to query the data content of the positioning condition from the on-premises database, i.e., the source data. The source data obtained at this time represents the final state of the data at the current moment. Then, this final state data is synchronized to the cloud database to ensure the eventual consistency of data between the cloud and on-premises. The following situations occur when parsing the positioning condition of the incremental SQL and querying the on-premises source data, as shown in Table 2:

[0126] Table 2 summarizes the results of querying cloud-based data based on incremental parsing positioning conditions.

[0127]

[0128] As shown in Table 2, cloud-based queries based on the location conditions parsed from incremental operation data have multiple possibilities, and the corresponding synchronization strategies vary depending on the query results. These situations will be elaborated upon below.

[0129] As shown in Table 2, when the location condition a = α, obtained from the incremental operation SQL, is not found in the on-premises database, there are two corresponding situations.

[0130] Scenario 1 (Table 2 Type No. ①): This data is no longer available on-premises. In this case, the data in the cloud database has two possible states: 1) The data may have been synchronized to the cloud before the current incremental SQL query was detected; 2) The data has not yet been migrated to the cloud, and the cloud database does not contain this data. However, regardless of the data's state in the cloud, according to the eventual consistency model, the data with the location condition a = α must be synchronously deleted in the cloud.

[0131] Scenario 2 (Table 2 Type No. ②): This incremental SQL statement is a special SQL operation, modifying the same field as the location condition field. In this case, if the data for a=α cannot be found on-premises, according to the eventual consistency model, the data with the a=α location condition on the cloud needs to be synchronously deleted. Then, a query is performed with a=β as the location condition. Based on the eventual consistency model, the results of the query with a=β as the location condition are synchronized to the cloud database. If the query for a=β yields results on-premises, this data is synchronously migrated to the cloud; if there are no results on-premises (possibly because a new incremental SQL statement modified the value of the a field after this SQL modification, or deleted the a=β data), the data with the a=β condition on the cloud (regardless of whether it exists on the cloud) is synchronously deleted.

[0132] As shown in Table 2, when results can be found in the cloud database based on the positioning condition a = α, there are still two cases.

[0133] Scenario 2 (Table 2 Type No. ②): This incremental SQL statement is a special SQL operation, modifying the same field as the location condition field. If this data is found in the on-premises database using a=α (possibly after this modification statement, a new incremental SQL statement inserted data with a=α), according to the eventual consistency model, the data with the location condition a=α in the cloud needs to be synchronously migrated to the cloud. Then, if a=β is used as the location condition for a query, according to the eventual consistency model, the result of the query with a=β as the location condition is synchronously migrated to the cloud database. If data with a=β is found in the on-premises database, this data is synchronously migrated to the cloud; if it does not exist in the on-premises database, the data with the condition a=β in the cloud (regardless of whether it exists in the cloud) is synchronously deleted.

[0134] Scenario 3 (Table 2 Type No. ③): The queried on-premises data represents the final state of this data at the current moment. Synchronizing this data to the cloud database ensures eventual consistency between the cloud and on-premises databases. During synchronization, note that the data in the cloud database may have two states: 1) This data does not yet exist in the cloud; simply insert it into the cloud database. 2) This data has already been migrated to the cloud database before parsing this incremental SQL statement; the current incremental SQL statement only modifies some of its attributes. In this case, the data in the cloud only needs to update the modified field values ​​in the on-premises database.

[0135] Therefore, this embodiment uses the `replace` statement when synchronously migrating on-premises data to the cloud database. If the data does not exist in the cloud, it is inserted; otherwise, all fields are directly replaced with the latest data values. The statement is shown below:

[0136] replace into table(a,b,c)values(α,β,γ);

[0137] The analysis of scenarios one, two, and three leads to the following conclusions: Figure 6 The cloud synchronization strategy is as follows: First, based on the positioning condition a = α obtained from the incremental SQL, query the on-premises database. If the query returns no results, the data with the positioning condition a = α in the cloud is synchronously deleted. If the query returns results, this data with a = α is synchronously migrated to the cloud database to ensure data consistency between the cloud and on-premises databases. Then, determine whether the current incremental SQL is a special SQL as described in Case Two. If it is not, the process ends. If it is a special SQL, modify the positioning condition to a = β and query the source data from the on-premises database. If a result is found, it is synchronously migrated to the cloud; if no result is found, the data with a = β in the cloud is synchronously deleted.

[0138] In the design of the traffic switching time point mentioned above, the selected switching time point T2 in this embodiment depends on the completion status of the data tracking program. Therefore, during the data tracking process, it is necessary to monitor the data migration progress in real time to determine the traffic switching time. In the implementation process, this embodiment scans the data volume of the source table on the cloud and the target table on the cloud in real time, checks the differences between the two, and prints them out in the form of logs. When the data tracking program obtains less than one batch of incremental table data, it can be considered that the data has been basically caught up, and at this time, a unified traffic switch is performed.

[0139] The data migration process involves a large amount of data, requiring strict adherence to data accuracy and successful completion. After migration, all on-premises and cloud traffic was stopped in the test environment, rendering both the source and target databases static and resistant to further changes. Automated scripts were then used to compare data entries one by one, supplemented by manual spot checks for cross-validation. Through multiple verifications of the data content and continuous improvement of the program, the accuracy of the migration process was ensured.

[0140] In a production environment, since this embodiment involves a non-stop data migration, both the cloud and on-premises systems are inevitably receiving traffic and modifying data. The data is not static, making content comparison difficult. Therefore, the main data inspection methods used in the production environment include: comparing the data volume between the old and new databases to see if there are any significant differences; monitoring system anomalies, focusing on the integrity of the transaction chain, the success rate of transactions, and investigating the reasons for failed transactions; and performing business verification on the system to ensure that transaction results meet expectations.

[0141] The traffic switching process is implemented in two parts: forwarding traffic from the original system and switching the consumer's address. These two parts work together to achieve complete traffic migration without generating any intermediate business data, thus achieving the goal of seamless switching for system users.

[0142] First, it's necessary to determine the timing of traffic forwarding. The data migration progress monitoring function is included in the data tracking program. This monitoring ensures real-time control over the data migration process. The program retrieves incremental SQL data from the incremental table in batches and modifies the consumption status of this incremental SQL data. When the number of incremental SQL data batches retrieved is less than the size of a complete batch, the data migration is considered essentially complete. At this point, the cloud data tracking switch and traffic forwarding switch should be turned on.

[0143] Then, the on-premises system uses a filter to forward all incoming traffic to the cloud system, preventing it from being processed for business logic in the on-premises system. This effectively avoids the risk of intermediate business data being generated due to simultaneous responses to business requests from both the cloud and on-premises systems.

[0144] Once traffic enters the cloud system, when business logic requests write operations to the database, the cloud uses a database interceptor to block them. It determines whether the incremental data has caught up by querying whether all data in the on-premises incremental table has been consumed. If the data has caught up, meaning all data in the incremental table has been consumed, the cloud traffic can proceed with write operations; otherwise, the cloud system rejects write operations.

[0145] After the on-premises system forwards traffic to the cloud system, each consumer will switch its calling address within a pre-defined timeframe. As the addresses of each consumer change, all system traffic will gradually switch to the new cloud system. The implementation time for each consumer's change can be determined based on the maintainability of the consumer's system; the switch only needs to be completed within a certain timeframe.

[0146] The above describes a data migration method provided by an embodiment of this application. The following describes the apparatus for performing the above data migration method.

[0147] This application provides a data migration apparatus, comprising:

[0148] The first migration unit is used to migrate the existing data in the source database before the first time point to the target database.

[0149] An interception unit is used to record incremental operations on the source database after the first time point into an incremental table using a database interceptor.

[0150] The second migration unit is used to migrate the incremental data corresponding to the incremental operation in the incremental table to the target database after all the existing data has been migrated to the target database.

[0151] The determining unit is used to determine a second time point based on the migration progress of the incremental data;

[0152] The flow-cutting unit is used to forward traffic accessing the source database to the target database at the second time point.

[0153] In one possible implementation, the first migration unit is specifically used to obtain the existing data to be migrated in the order of the primary keys of the data in the source database; determine the physical shard of the existing data to be migrated in the target database according to the sharding rules of the data table corresponding to the existing data to be migrated; and call each physical shard processing thread to write the existing data to be migrated to the corresponding physical shard using a batch submission method.

[0154] In one possible implementation, the first migration unit is further configured to: acquire data migration status when an abnormal exit is detected; acquire breakpoint data location information based on the data migration status; and trigger the execution of the step of acquiring the existing data to be migrated in sequence based on the primary key of the data in the source database, based on the breakpoint data location information, until all the existing data is migrated to the target database.

[0155] In one possible implementation, the second migration unit is specifically used to obtain incremental operation data recorded in the incremental table; parse the incremental operation data to obtain the location conditions of the incremental operation data in the source database; obtain the incremental data to be migrated according to the location conditions; and migrate the incremental data to be migrated to the target database.

[0156] In one possible implementation, the determining unit is specifically used to determine the current time point as the second time point when there is less than one batch of remaining incremental data to be migrated.

[0157] In one possible implementation, the flow-cutting unit is specifically used to forward traffic accessing the source database to the target database at the second time point using the filter of the source database; to intercept write operations to the target database using a database interceptor; and to execute write operations to the target database if all incremental data corresponding to all incremental operations in the incremental table have been migrated to the target database; and to refuse to execute write operations to the target database if not all incremental data corresponding to all incremental operations in the incremental table have been migrated to the target database.

[0158] This application also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0159] The memory is used to store computer programs;

[0160] The processor is used to execute the computer program so that the electronic device can implement any of the data migration methods provided in the embodiments of this application.

[0161] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the data migration methods provided in this application.

[0162] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the data migration methods provided in this application.

[0163] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0165] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0166] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data migration method, characterized in that, include: Migrate existing data in the source database prior to the first time point to the target database; A database interceptor is used to record incremental operations on the source database after the first time point to an incremental table; After all the existing data has been migrated to the target database, the incremental data corresponding to the incremental operations in the incremental table will be migrated to the target database. The second time point is determined based on the migration progress of the incremental data; At the second time point, traffic accessing the source database is forwarded to the target database.

2. The data migration method according to claim 1, characterized in that, The process of migrating existing data from the source database prior to the first time point to the target database includes: The existing data to be migrated is obtained based on the order of the primary keys in the source database; Based on the sharding rules of the data table corresponding to the existing data to be migrated, determine the physical sharding of the existing data to be migrated in the target database; The batch submission method is used to call each physical shard processing thread to write the existing data to be migrated into the corresponding physical shard.

3. The data migration method according to claim 2, characterized in that, The process of migrating existing data from the original database prior to the first time point to the target database also includes: If an abnormal exit is detected, obtain the data migration status; Obtain the breakpoint data location information based on the data migration status; The step of obtaining the existing data to be migrated in sequence based on the data primary key in the source database is triggered according to the breakpoint data location information, until all the existing data is migrated to the target database.

4. The data migration method according to claim 1, characterized in that, The step of migrating the incremental data corresponding to the incremental operation in the incremental table to the target database includes: Retrieve the incremental operation data recorded in the incremental table; The incremental operation data is parsed to obtain the location conditions of the incremental operation data in the source database; Obtain the incremental data to be migrated based on the positioning conditions; The incremental data to be migrated is migrated to the target database.

5. The data migration method according to claim 1, characterized in that, Determining the second time point based on the migration progress of the incremental data includes: If there is less than one batch of incremental data remaining to be migrated, the current time point will be designated as the second time point.

6. The data migration method according to claim 1, characterized in that, Forwarding the traffic accessed by the source database to the target database at the second time point includes: At the second time point, the source database's filter is used to forward traffic accessing the source database to the target database; Use a database interceptor to intercept write operations to the target database; If all incremental data corresponding to all incremental operations in the incremental table has been migrated to the target database, then a write operation is performed on the target database. If not all incremental data corresponding to all incremental operations in the incremental table has been migrated to the target database, then write operations to the target database will be refused.

7. A data migration device, characterized in that, include: The first migration unit is used to migrate the existing data in the source database before the first time point to the target database. An interception unit is used to record incremental operations on the source database after the first time point into an incremental table using a database interceptor. The second migration unit is used to migrate the incremental data corresponding to the incremental operation in the incremental table to the target database after all the existing data has been migrated to the target database. The determining unit is used to determine a second time point based on the migration progress of the incremental data; The flow-cutting unit is used to forward traffic accessing the source database to the target database at the second time point.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the data migration method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the data migration method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the data migration method as described in any one of claims 1 to 6.

Images