Data migration method and device, nonvolatile storage medium and computer equipment

Abstract

The invention discloses a data migration method and device, a nonvolatile storage medium and computer equipment. The method comprises the following steps: receiving a migration instruction; based on the migration indication, starting to record a transaction operation on the source database and determining to-be-migrated data in the source database; storing the to-be-migrated data into a target database; when the storage of the to-be-migrated data in the target database is finished, suspending the transaction operation of recording the source database to obtain a target transaction record; and based on the target transaction record, adjusting data in the target database to obtain target data in the target database. The technical problems that when data migration is carried out at present, updating of a source database needs to be paused, service continuity is affected, and stability and reliability of a system are reduced are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a data migration method and device, a nonvolatile storage medium and a computer device. BACKGROUND

[0002] With the rapid development of information technology, the scale of data is growing explosively, and there are problems such as performance bottleneck of old databases and architecture landing. Enterprises and organizations urgently need to migrate to new databases to improve data processing efficiency and storage capacity. Traditional database migration schemes usually have the following limitations:

[0003] 1. Business interruption risk: The current data migration in the database needs to export and import all data under the condition that the source database stops serving, which often leads to long downtime of the business system, seriously affecting business continuity. In today's enterprise environment that relies heavily on real-time data processing, any downtime can cause significant economic losses and customer experience decline.

[0004] 2. Challenge of data consistency: The current data migration method also has the problem of real-time synchronization of incremental data through database log analysis and distributed transactions. However, this approach increases the complexity and coupling of the system. The success of a single transaction is highly dependent on the simultaneous success of operations on both databases. Once any database fails, the transaction may fail, seriously threatening data consistency and system reliability.

[0005] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0006] The embodiments of the present application provide a data migration method, device, nonvolatile storage medium and computer device to at least solve the technical problem that the update of the source database needs to be suspended during data migration, affecting the continuity of the business, reducing the stability and reliability of the system.

[0007] According to an aspect of an embodiment of the present application, a data migration method is provided, comprising: receiving a migration instruction; based on the migration instruction, starting to record transaction operations on a source database and determining to-be-migrated data in the source database; storing the to-be-migrated data into a target database; in the case that the to-be-migrated data is stored into the target database, suspending the recording of the transaction operations on the source database to obtain target transaction records; based on the target transaction records, adjusting the data in the target database to obtain target data in the target database.

[0008] Optionally, storing the data to be migrated into the target database includes: obtaining the target data format corresponding to the target database; modifying the target data format to obtain the modified data to be migrated; and storing the modified data to be migrated into the target database.

[0009] Optionally, based on the target transaction records, the data in the target database is adjusted to obtain the target data in the target database, including: determining the data to be adjusted based on the target transaction records; determining whether the data to be adjusted is located in the data to be migrated; if the data to be adjusted is located in the data to be migrated, adjusting the data in the target database to obtain the target data in the target database; and / or, if the data to be adjusted is not located in the data to be migrated, adjusting the data in the source database.

[0010] Optionally, based on the target transaction records, the data in the target database is adjusted to obtain the target data in the target database, including: adjusting the data in the source database and the target database based on the target transaction records to obtain the adjustment results of the source database and the target database; if the adjustment result of the source database is successful and the adjustment result of the target database is unsuccessful, a positive compensation log is obtained, wherein the positive compensation log includes the reason for the failure of the target database adjustment and the database location; the adjusted data in the source database is obtained; based on the positive compensation log and the adjusted data in the source database, the data in the target database is repaired to determine the target data in the target database.

[0011] Optionally, based on the target transaction records, the data in the source database and the target database are adjusted to obtain the source database adjustment results and the target database adjustment results. This includes: adjusting the data in the source database and the target database based on the target transaction records, and monitoring the adjustment process of the source database and the target database; if an anomaly occurs during the adjustment process of the source database, writing the reason for the source database adjustment failure and the database location to a negative compensation log; if an anomaly occurs during the adjustment process of the target database, writing the reason for the target database adjustment failure and the database location to a positive compensation log; after the data adjustment in the source database and the target database is completed, comparing the data in the source database and the data in the target database to obtain a comparison result; and based on the comparison result, determining the source database adjustment results and the target database adjustment results.

[0012] Optionally, it also includes: if the adjustment result of the source database is adjustment failure and the adjustment result of the target database is adjustment success, obtaining the reverse compensation log and determining the target data in the target database; based on the reverse compensation log and the target data in the target database, repairing the data in the source database, and determining the repaired data in the source database.

[0013] According to another aspect of the present invention, a data migration apparatus is also provided, comprising: a receiving module for receiving a migration instruction; a determining module for starting to record transaction operations on a source database and determining data to be migrated in the source database based on the migration instruction; a storing module for storing the data to be migrated into a target database; a pausing module for pausing the recording of transaction operations on the source database after the storage of the data to be migrated into the target database is completed, thereby obtaining a target transaction record; and an adjusting module for adjusting the data in the target database based on the target transaction record, thereby obtaining target data in the target database.

[0014] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the device where the non-volatile storage medium is located to execute any of the above data migration methods.

[0015] According to another aspect of the present invention, a computer device is also provided, the computer device including a processor, the processor being configured to run a program, wherein the program executes any of the above-described data migration methods during runtime.

[0016] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described data migration methods.

[0017] In this embodiment of the invention, a data migration method is employed. This involves receiving a migration instruction; based on the instruction, starting to record transaction operations on the source database and identifying the data to be migrated in the source database; storing the data to be migrated into the target database; pausing the recording of transaction operations in the source database after the data migration is complete, thus obtaining the target transaction record; and adjusting the data in the target database based on the target transaction record to obtain the target data. This achieves the goal of continuing business operations in the source database during data migration, thereby ensuring business continuity and improving system stability and reliability. It also solves the technical problem that currently, data migration requires pausing source database updates, affecting business continuity and reducing system stability and reliability. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1A hardware structure block diagram of a computer terminal for implementing a data migration method is shown.

[0020] Figure 2 This is a flowchart illustrating the data migration method provided according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the overall framework of the apparatus for the data migration method provided in an optional embodiment of the present invention;

[0022] Figure 4 This is a flowchart illustrating the operation of a data migration method provided by an optional embodiment of the present invention;

[0023] Figure 5 This is a structural block diagram of a data migration apparatus provided according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to an embodiment of the present invention, a method embodiment of a data migration method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1A hardware block diagram of a computer terminal for implementing a data migration method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data migration method in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the data migration method of the application described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0031] Figure 2 This is a flowchart illustrating the data migration method provided in an embodiment of the present invention, as shown below.Figure 2 As shown, the method includes the following steps:

[0032] Step S202: Receive migration instruction.

[0033] Optionally, migration instructions typically originate from an enterprise's IT decision-making level. These instructions may stem from business development needs, technology upgrades, performance optimizations, or architectural adjustments, leading to a decision to migrate from an existing source database (such as a traditional relational database) to a new target database (such as a cloud database or distributed database). This instruction can be passed to the data migration system via command-line parameters, configuration file updates, or API calls.

[0034] The migration details should include the following information: the database type, version, address, and login credentials of both the source and target databases; the migration time window, which defines the permitted time period for database migration. Considering business continuity and minimizing the impact on online services, this is typically done during off-peak business periods or maintenance windows; the scope of data to be migrated, which may be all data, specific tables, specific datasets, or subsets of data meeting certain conditions; migration priorities to coordinate resource allocation; and any special migration requirements, such as data cleaning rules, data format conversion, and post-migration data consistency checks.

[0035] Specifically, after receiving a migration instruction, a data migration system typically has a feedback mechanism to confirm whether the system is ready to begin the migration. This involves connection testing with the source and target databases, resource availability checks, and preliminary review of the migration scripts. Once the system confirms its readiness, it sends a confirmation signal to the instruction initiator, indicating that the data migration is about to begin. Because database migration often involves adjustments to sensitive data and core business processes, strict security controls should be in place for the process of receiving migration instructions, such as access control and storage access logs.

[0036] Step S204: Based on the migration instructions, start recording transaction operations on the source database and determine the data to be migrated in the source database.

[0037] Optionally, upon receiving a migration instruction, it's time to begin data migration. Therefore, it's necessary to start recording transaction operations on the source database without modifying the database content. This avoids problems caused by real-time database modifications; recording transaction operations allows for data modification after the migration is complete.

[0038] Specifically, initiating transaction recording should include the following operations: initializing the transaction record cache; upon receiving the migration instruction, the program control unit immediately turns on the transaction record switch, activating the transaction record cache. The transaction record cache is used to store records of all transaction operations performed on the source database, including but not limited to data insertion, update, and deletion operations. With the transaction record switch on, the data migration system automatically captures and records the complete details of each transaction operation. This information includes, but is not limited to: the type of operation (e.g., update, insert), the data ID involved in the operation, the data state before and after the operation, the timestamp of the operation, and migration flag bits, for subsequent comparison and compensation operations.

[0039] Specifically, determining the data to be migrated from the source database includes the following operations: Simultaneously with initiating the transaction record process, it is also necessary to determine which data in the source database is the target for migration. This is based on the data range information in the migration instruction, i.e., the specific range given in step S202. The existing data migration unit will export snapshots of the data to be migrated from the source database according to the determined data range. These data snapshots contain the current state of all data within the specified range, used for subsequent data migration and comparison. Before importing the data snapshots into the target database, data format conversion is required to ensure that the data can adapt to the structural requirements of the target database. Data format conversion may be performed through a built-in or custom data format converter to ensure data compatibility and accuracy during the migration process.

[0040] This invention introduces a closed-loop refresh unit and a hot refresh mechanism. The data migration process is not static but dynamically adjusted based on system operation and migration progress. The migration data selector in the program control unit uses the concept of a closed-loop data unit to determine the data migration direction. A closed-loop data unit is a set of data where all transactions in the application strongly depend on only one side of the database for their operations on the data within that set; the results of operations on the other side of the database do not affect the transaction results. Furthermore, to support the dynamic adjustment of the closed-loop data unit, the data migration system employs a hot refresh mechanism, allowing for real-time updates and adjustments to the data migration scope even during the migration process, without requiring application restarts or service interruptions. This flexible adjustment mechanism enables smooth data migration, gradually expanding the participation of the target database until the migration of all data is finally completed.

[0041] Step S206: Store the data to be migrated into the target database.

[0042] Optionally, storing the data to be migrated into the target database may include the following steps:

[0043] Step S206.1: Perform preprocessing before data import, such as checking the integrity of the data to ensure that each record has the necessary field values; performing data cleaning to remove invalid or redundant data; and performing necessary data conversions, such as converting the currency unit of numeric types from US dollars to RMB.

[0044] Step S206.2: Officially store the modified data to be migrated into the target database. This step involves batch data insertion, update, or deletion operations, which should be flexibly selected based on the specific data migration requirements and the characteristics of the target database.

[0045] Step S206.3: If an error occurs during the data import process into the target database, such as data type mismatch, primary key conflict, or violation of constraints, the data import should be stopped immediately. Detailed error information should be recorded, and appropriate error handling measures should be taken, such as data correction, re-import, or rollback, to ensure data integrity and consistency. Simultaneously, these error messages should be logged for later analysis and tracking. If no errors occur, skip this step.

[0046] Step S206.4: After the data import is complete, data verification is required. This can be done by comparing the hash values ​​and statistical values ​​(such as the number of records, total amount, etc.) of the same data in the source database and the target database. If data inconsistencies are found, the data consistency repair process needs to be initiated to ensure data consistency.

[0047] Step S208: After the data to be migrated is stored in the target database, pause the transaction operation of recording the source database to obtain the target transaction record.

[0048] In this step, once the migration of existing data to the target database is complete, to ensure data consistency between the source and target databases, the recording of transaction operations in the source database can be paused. This means that no new data change records are recorded for a certain period. A target transaction record is generated and saved by collecting the transaction logs from the source database before the recording pause. The transaction logs contain modification information for each transaction, such as the modified data ID, the data values ​​before and after the modification, the modification timestamp, and the modification type. The target transaction record contains details of all transaction operations in the source database within the pause time window.

[0049] The time window for suspending transaction records should be set reasonably in terms of both time period and length. This operation can be carried out during off-peak business hours, and the length of the time window should be sufficient to capture all possible transaction operations while minimizing business interruption time.

[0050] Step S210: Based on the target transaction records, adjust the data in the target database to obtain the target data in the target database.

[0051] This step involves parsing the target transaction records obtained from the source database when recording was paused. These target transaction records contain all data change operations that occurred on the source database during the data migration, including but not limited to inserts, updates, and deletions. The parsing process primarily identifies the specific operation type and involved data objects for each transaction. Through the parsed transaction records, the data requiring correction is located. It is then determined whether the data requiring correction is located within the data to be migrated. If the data to be adjusted is located within the data to be migrated, this means that this data has already been processed during the data migration phase. In this case, the data in the target database is adjusted to ensure consistency with the changes in the source database. If the data to be adjusted is not located within the data to be migrated, the relevant data in the source database should be adjusted to maintain data synchronization.

[0052] By following the steps above, the business continuity and reliability of the application during the database migration process can be guaranteed, thereby solving the technical problem that currently, when data migration is carried out, it is necessary to pause the updates of the source database, which affects the business continuity and reduces the stability and reliability of the system.

[0053] As an optional embodiment, storing the data to be migrated into the target database includes: obtaining the target data format corresponding to the target database; modifying the target data format to obtain the modified data to be migrated; and storing the modified data to be migrated into the target database.

[0054] Optionally, you can first identify the type of the target database and its specific data storage format. This allows you to convert the data format later to meet the requirements of the target database. This can be done by querying the target database's system directory or using the API provided by the database management software to obtain information such as the target database's table structure and field types.

[0055] Obtaining the target data format corresponding to the target database ensures that the data can be correctly imported into the target database, achieving consistency and compatibility in data format when migrating between different databases. This process ensures seamless data integration with the target database, avoiding migration failures due to data format incompatibility and improving the efficiency and success rate of data migration.

[0056] As an optional embodiment, adjusting the data in the target database based on the target transaction records to obtain the target data in the target database includes: determining the data to be adjusted based on the target transaction records; determining whether the data to be adjusted is located in the data to be migrated; if the data to be adjusted is located in the data to be migrated, adjusting the data in the target database to obtain the target data in the target database; and / or, if the data to be adjusted is not located in the data to be migrated, adjusting the data in the source database.

[0057] Optionally, based on the target transaction record, the incremental data migration unit first parses each change instruction in the target transaction record to identify the data ID and change type (such as insert, update or delete). The incremental data migration unit calls the route resolver to confirm whether the data item is already within the range of data to be migrated based on the data ID and change type.

[0058] Once it's confirmed that the data to be adjusted is located within the data to be migrated, the data migration unit performs the corresponding data adjustment operations on the target database using the data migrater. For example, if the target transaction record indicates an update operation, the data migrater will read the latest state of that data item in the source database and then perform the same update operation in the target database, ensuring that the data states of the two databases remain consistent. If the operation is successful, the data migrater will update the migration flag in the transaction record to indicate "migrated." If the operation fails, the data migrater will record the failure event in the positive compensation log, which will then be processed by the positive data compensation unit to ensure data consistency and integrity.

[0059] As an optional embodiment, based on the target transaction record, the data in the target database is adjusted to obtain the target data in the target database, including: adjusting the data in the source database and the target database based on the target transaction record to obtain the adjustment result of the source database and the adjustment result of the target database; if the adjustment result of the source database is successful and the adjustment result of the target database is unsuccessful, a positive compensation log is obtained, wherein the positive compensation log includes the reason and database location corresponding to the failure of the target database adjustment; the adjusted data in the source database is obtained; based on the positive compensation log and the adjusted data in the source database, the data in the target database is repaired to determine the target data in the target database.

[0060] Optionally, the system can parse target transaction records containing all data modification operations performed on the source database within the time window of the paused record's transaction operations. These operations include inserts, updates, and deletions. The parsing process identifies the specific operation type and the data objects involved in each transaction. Then, based on these parsed transaction records, corresponding data adjustment operations are performed simultaneously in both the source and target databases. After the data adjustment operations are completed, the adjustment results in the source and target databases are compared to determine if the adjustment was successful. If the comparison shows that the adjustment result in the source database is successful while the adjustment result in the target database is unsuccessful, repair is required.

[0061] Specifically, consider the migration of core data from a source database to a target database. After the existing data migration is complete, transaction records in the source database are paused to allow for incremental data synchronization. Due to differences in data types and constraints between the target and source databases, a data type mismatch error occurs during the incremental data synchronization process, causing the update to fail. The positive compensation log records this error information, including the error type (data type mismatch), the data ID involved (customer account ID), and the timestamp of the error. The system then retrieves the successfully updated information from the source database, converts the data type to a format compatible with the target database, performs a repair operation in the target database, and finally updates the record, ensuring data consistency between the two databases.

[0062] As an optional embodiment, based on the target transaction records, the data in the source database and the target database are adjusted to obtain the source database adjustment result and the target database adjustment result. This includes: adjusting the data in the source database and the target database based on the target transaction records, and monitoring the adjustment process of the source database and the target database; if an anomaly occurs during the adjustment process of the source database, writing the reason for the source database adjustment failure and the database location to a negative compensation log; if an anomaly occurs during the adjustment process of the target database, writing the reason for the target database adjustment failure and the database location to a positive compensation log; after the data adjustment in the source database and the target database is completed, comparing the data in the source database and the data in the target database to obtain a comparison result; and based on the comparison result, determining the source database adjustment result and the target database adjustment result.

[0063] Optionally, a negative compensation log can be introduced. When the data migration system encounters problems such as data format mismatch, primary key conflict, network latency, or temporary database failures while attempting to synchronize data from the source database to the target database, these failed synchronization attempts should be recorded in the negative compensation log. This includes, but is not limited to, the failed data ID, data value, operation type (such as insert, update, or delete), and the specific reason for the failure (such as database constraint violation, type mismatch, etc.). Specifically, when forward data synchronization (i.e., synchronization from the source database to the target database) fails, the system will write relevant information to the negative compensation log. The data compensation unit will read these logs and attempt to repair the data in the target database based on the latest state of the source database to achieve data consistency between the two databases. If the data in the source database has been changed after the synchronization failure, the data compensation unit will perform corresponding operations based on the latest source database data.

[0064] Figure 3 This is a schematic diagram of the overall framework of the device for the data migration method provided in an optional embodiment of the present invention, as shown below. Figure 3 As shown, transaction flow data is input to the program control unit, which includes three sub-modules: a record controller, a dual-write controller, and a migration data selector. The program control unit is connected to the forward data compensation unit, the reverse data compensation unit, and the incremental data migration unit. The incremental data migration unit and the existing data migration unit work together on the source database and the target database. The incremental data migration unit includes modules such as a route resolver, a data synchronizer, a data comparer, an anomaly handler, and a cache. It also includes a data comparison unit that interacts directly with the source database, and an anomaly monitoring unit that monitors the target database. These components together constitute the apparatus for the data migration method.

[0065] Figure 4 This is a flowchart illustrating the operation of a data migration method provided by an optional embodiment of the present invention, such as... Figure 4As shown, the specific operational logic is presented based on the connection architecture. Specifically, after the program control unit receives the migration instruction, the transaction recording switch is activated, ensuring that any write operations to the source database are recorded in the transaction record cache. The recorded content includes, but is not limited to: the operation's unique identifier ID, data details, operation timestamp, migration flag, and change type. The existing data migration unit exports all historical data from the source database and converts this existing data to a format suitable for the target database. The program control unit switches the dual-write switch, stopping transaction recording. If a write operation to the target database fails, the failure information is recorded in the positive compensation log. The incremental data migration unit extracts all data changes generated and recorded after the existing data migration based on the recorded transaction records; this incremental data is synchronized to the target database. The program control unit initiates the positive compensation unit, checks the positive compensation log, and searches for all write operation records that failed to execute successfully in the target database. For each record, this unit retrieves the current state of the corresponding data from the source database and attempts to synchronize this data to the target database again. After the forward compensation unit completes its work, the data comparison unit performs a full data comparison to verify data consistency during the database migration process. For inconsistencies, this unit performs a reasonableness assessment, such as checking timestamps, forward compensation logs, and reverse compensation logs. The program control unit adjusts the scope of the data migration based on business requirements and system performance metrics, switching the flow of some data to the target database. Write operations on this data are first performed in the target database and then in the source database. If a write operation in the source database fails, the error message is recorded in the reverse compensation log. The program control unit then initiates the reverse compensation unit, checks the reverse compensation log, and identifies the write operation records that failed to execute successfully in the source database. For each record, this unit retrieves the current state of the corresponding data from the target database and attempts to synchronize this data to the source database. After confirming that all data has been smoothly migrated to the target database, the dual-write switch is turned off, and the data migration operation is complete.

[0066] As an optional embodiment, it further includes: when the adjustment result of the source database is adjustment failure and the adjustment result of the target database is adjustment success, obtaining the reverse compensation log and determining the target data in the target database; based on the reverse compensation log and the target data in the target database, repairing the data in the source database, and determining the repaired data in the source database.

[0067] When a source database adjustment operation fails while the target database adjustment operation succeeds, the data changes have taken effect in the target database, but the source database update was not completed. To maintain data consistency, all change operations that occurred in the target database but failed to execute successfully in the source database need to be recorded in the reverse compensation log. Repair operations are then performed based on the reverse compensation log and the target data in the target database to ensure data consistency between the source and target databases. The repair operation includes the following steps: using components such as a data synchronizer and a repair executor, the data in the source database is repaired based on information in the reverse compensation log to ensure its state matches the target data state in the target database. Further, components such as a data comparator and a consistency checker are used to compare the data and verify the success of the repair. If the repair operation is successful, the compensation flag in the relevant record in the reverse compensation log is updated to "completed"; if the repair operation fails, a detailed reason for the failure is recorded, and an alarm mechanism is triggered so that operations personnel can intervene manually.

[0068] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the data migration method according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0070] According to an embodiment of the present invention, a data migration apparatus for implementing the above-described data migration method is also provided. The figure is a structural block diagram of the data migration apparatus provided according to an embodiment of the present invention. Figure 5 As shown, the data migration device includes: a receiving module 502, a determining module 504, a storing module 506, a pausing module 508, and an adjusting module 510. The data migration device will be described below.

[0071] The receiving module 502 is used to receive migration instructions.

[0072] The determination module 504 is used to record transaction operations on the source database and determine the data to be migrated in the source database.

[0073] The storage module 506 is used to store the data to be migrated into the target database.

[0074] The pause module 508 is used to pause the transaction operation of the source database and obtain the target transaction record after the data to be migrated has been stored in the target database.

[0075] The adjustment module 510 is used to adjust the data in the target database based on the target transaction records to obtain the target data in the target database.

[0076] It should be noted that the receiving module 502, determining module 504, storing module 506, pausing module 508, and adjusting module 510 correspond to steps S202 to S210 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in the embodiments.

[0077] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0078] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data migration method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned data migration method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] The processor can invoke information and application programs stored in the memory via the transmission device to perform the following steps: receiving a migration instruction; based on the migration instruction, starting to record transaction operations on the source database and determining the data to be migrated in the source database; storing the data to be migrated into the target database; when the storage of the data to be migrated into the target database is completed, pausing the recording of transaction operations on the source database to obtain the target transaction record; and adjusting the data in the target database based on the target transaction record to obtain the target data in the target database.

[0080] Optionally, the processor may also execute program code that performs the following steps: storing the data to be migrated into the target database, including: obtaining the target data format corresponding to the target database; modifying the target data format to obtain the modified data to be migrated; and storing the modified data to be migrated into the target database.

[0081] Optionally, the processor may also execute program code that performs the following steps: adjusting data in the target database based on the target transaction record to obtain target data in the target database, including: determining the data to be adjusted based on the target transaction record; determining whether the data to be adjusted is located in the data to be migrated; adjusting the data in the target database if the data to be adjusted is located in the data to be migrated to obtain target data in the target database; and / or adjusting the data in the source database if the data to be adjusted is not located in the data to be migrated.

[0082] Optionally, the processor may also execute program code for the following steps: adjusting data in the target database based on the target transaction record to obtain target data in the target database, including: adjusting data in the source database and the target database based on the target transaction record to obtain adjustment results for the source database and the target database; if the adjustment result of the source database is successful and the adjustment result of the target database is unsuccessful, obtaining a positive compensation log, wherein the positive compensation log includes the reason for the failure of the target database adjustment and the database location; obtaining the adjusted data in the source database; and repairing the data in the target database based on the positive compensation log and the adjusted data in the source database to determine the target data in the target database.

[0083] Optionally, the processor may also execute program code that performs the following steps: adjusting the data in the source database and the target database based on the target transaction record to obtain the source database adjustment result and the target database adjustment result, including: adjusting the data in the source database and the target database based on the target transaction record and monitoring the adjustment process of the source database and the target database; writing the reason for the source database adjustment failure and the database location to a negative compensation log if an anomaly occurs during the source database adjustment process; writing the reason for the target database adjustment failure and the database location to a positive compensation log if an anomaly occurs during the target database adjustment process; comparing the data in the source database and the target database after the data adjustment in the source database and the target database is completed to obtain a comparison result; and determining the source database adjustment result and the target database adjustment result based on the comparison result.

[0084] Optionally, the processor may also execute program code that performs the following steps: if the source database adjustment result is adjustment failure and the target database adjustment result is adjustment success, obtain the reverse compensation log and determine the target data in the target database; based on the reverse compensation log and the target data in the target database, repair the data in the source database and determine the repaired data in the source database.

[0085] This invention provides a data migration method. The method involves receiving a migration instruction; based on the instruction, starting to record transaction operations on the source database and identifying the data to be migrated in the source database; storing the data to be migrated into the target database; pausing the recording of transaction operations in the source database after the data migration is completed, thus obtaining the target transaction record; and adjusting the data in the target database based on the target transaction record to obtain the target data in the target database. This achieves the goal of continuing business operations in the source database during data migration, thereby solving the technical problem in related technologies where pausing source database updates during data migration affects business continuity and reduces system stability and reliability.

[0086] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0087] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the data migration method provided in the above embodiments.

[0088] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0089] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: receiving a migration instruction; based on the migration instruction, starting to record transaction operations on the source database and determining the data to be migrated in the source database; storing the data to be migrated into the target database; when the storage of the data to be migrated into the target database is completed, pausing the recording of transaction operations on the source database to obtain the target transaction record; and adjusting the data in the target database based on the target transaction record to obtain the target data in the target database.

[0090] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: storing the data to be migrated into the target database, including: obtaining the target data format corresponding to the target database; modifying the target data format of the data to be migrated to obtain the modified data to be migrated; and storing the modified data to be migrated into the target database.

[0091] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: adjusting data in the target database based on the target transaction record to obtain target data in the target database, including: determining the data to be adjusted based on the target transaction record; determining whether the data to be adjusted is located in the data to be migrated; if the data to be adjusted is located in the data to be migrated, adjusting the data in the target database to obtain target data in the target database; and / or, if the data to be adjusted is not located in the data to be migrated, adjusting the data in the source database.

[0092] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: adjusting data in the target database based on the target transaction record to obtain target data in the target database, including: adjusting data in the source database and the target database based on the target transaction record to obtain source database adjustment results and target database adjustment results; if the source database adjustment result is successful and the target database adjustment result is unsuccessful, obtaining a positive compensation log, wherein the positive compensation log includes the reason for the target database adjustment failure and the database location; obtaining the adjusted data in the source database; and repairing the data in the target database based on the positive compensation log and the adjusted data in the source database to determine the target data in the target database.

[0093] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: adjusting data in the source database and the target database based on the target transaction record to obtain source database adjustment results and target database adjustment results, including: adjusting data in the source database and the target database based on the target transaction record and monitoring the adjustment process of the source database and the target database; if an anomaly occurs during the adjustment process of the source database, writing the reason for the source database adjustment failure and the database location to a negative compensation log; if an anomaly occurs during the adjustment process of the target database, writing the reason for the target database adjustment failure and the database location to a positive compensation log; after the data adjustment in the source database and the target database is completed, comparing the data in the source database and the data in the target database to obtain a comparison result; and determining the source database adjustment result and the target database adjustment result based on the comparison result.

[0094] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: if the source database adjustment result is adjustment failure and the target database adjustment result is adjustment success, obtain the reverse compensation log and determine the target data in the target database; based on the reverse compensation log and the target data in the target database, repair the data in the source database and determine the repaired data in the source database.

[0095] Embodiments of the present invention also provide a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can: receive a migration instruction; based on the migration instruction, start recording transaction operations on a source database and determine the data to be migrated in the source database; store the data to be migrated into a target database; when the storage of the data to be migrated into the target database is completed, pause the recording of transaction operations on the source database to obtain a target transaction record; and based on the target transaction record, adjust the data in the target database to obtain the target data in the target database.

[0096] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0097] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0102] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that, include: Receive migration instructions; Based on the migration instruction, start recording transaction operations on the source database and determine the data to be migrated in the source database; Store the data to be migrated into the target database; When the data to be migrated is stored in the target database, the transaction recording operation of the source database is paused, and the target transaction record is obtained; Based on the target transaction record, the data in the target database is adjusted to obtain the target data in the target database.

2. The method according to claim 1, characterized in that, The step of storing the data to be migrated into the target database includes: Obtain the target data format corresponding to the target database; The target data format is used to modify the data to be migrated, resulting in the modified data to be migrated. The modified data to be migrated is stored in the target database.

3. The method according to claim 1, characterized in that, The step of adjusting the data in the target database based on the target transaction record to obtain the target data in the target database includes: Based on the target transaction records, determine the data to be adjusted; Determine whether the data to be adjusted is located within the data to be migrated; If the data to be adjusted is located within the data to be migrated, the data in the target database is adjusted to obtain the target data in the target database; And / or, if the data to be adjusted is not located in the data to be migrated, the data in the source database is adjusted.

4. The method according to claim 1, characterized in that, The step of adjusting the data in the target database based on the target transaction record to obtain the target data in the target database includes: Based on the target transaction record, the data in the source database and the target database are adjusted to obtain the source database adjustment result and the target database adjustment result. If the source database adjustment result is successful and the target database adjustment result is unsuccessful, a positive compensation log is obtained, wherein the positive compensation log includes the reason and database location corresponding to the target database adjustment failure; Obtain the adjusted data from the source database; Based on the positive compensation log and the adjusted data in the source database, the data in the target database is repaired to determine the target data in the target database.

5. The method according to claim 4, characterized in that, The step of adjusting the data in the source database and the target database based on the target transaction record to obtain the source database adjustment result and the target database adjustment result includes: Based on the target transaction records, the data in the source database and the target database are adjusted, and the adjustment process of the source database and the target database is monitored. If an anomaly occurs during the adjustment of the source database, the reason for the source database adjustment failure and the database location will be written into the negative compensation log. If an anomaly occurs during the adjustment of the target database, the reason for the failure of the target database adjustment and the database location shall be written into the positive compensation log; Once the data adjustment in the source database and the target database is completed, the data in the source database and the data in the target database are compared to obtain the comparison result; Based on the comparison results, the adjustment results of the source database and the adjustment results of the target database are determined.

6. The method according to claim 4, characterized in that, Also includes: If the source database adjustment result is adjustment failure and the target database adjustment result is adjustment success, obtain the reverse compensation log and determine the target data in the target database; Based on the reverse compensation log and the target data in the target database, the data in the source database is repaired to determine the repaired data in the source database.

7. A data migration device, characterized in that, include: The receiving module is used to receive migration instructions; The determination module is used to start recording transaction operations on the source database and determine the data to be migrated in the source database based on the migration instruction. The storage module is used to store the data to be migrated into the target database; The pause module is used to pause the recording of transaction operations in the source database and obtain the target transaction records when the data to be migrated is stored in the target database. The adjustment module is used to adjust the data in the target database based on the target transaction record to obtain the target data in the target database.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the data migration method according to any one of claims 1 to 6.

9. A computer device, characterized in that, include: Memory and processor The memory stores computer programs; The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor performs the data migration method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data migration method according to any one of claims 1 to 6.

Images