Data migration method, device, and storage medium

Abstract

This application discloses a data migration method, apparatus, and storage medium, relating to the fields of data processing and database technology. The method includes: responding to a data migration instruction, acquiring first existing data and initial identification information of the first existing data in a source database; if initial identification information exists in a target database, acquiring first version information of the first existing data and second version information of data already written to the target database; and writing the first existing data to the target database when the data update sequence represented by the first version information is later than the data update sequence represented by the second version information. This application can ensure that the data in the target database is a newer version of the data while ensuring zero business downtime and zero awareness, thereby preventing older existing data from overwriting new data already written to the target database, and thus improving the eventual consistency of full data migration between the source and target databases.

Description

Technical Field

[0001] This application relates to the fields of data processing and database technology, and in particular to a data migration method, device and storage medium. Background Technology

[0002] As enterprises deepen their digital transformation, scenarios such as database data center migration and cloud transformation are increasing. MongoDB (Mongo Database), as a non-relational database, is widely used in various core business systems due to its flexible data model and high concurrency processing capabilities.

[0003] Currently, common MongoDB data migration methods typically send write requests to both the source and target databases simultaneously during the migration of existing data, aiming to achieve zero business downtime during the migration process. However, in complex scenarios where existing and incremental data are written in parallel, the order of completion of the two writes becomes uncertain when data with the same primary key is updated in real time during the existing data synchronization task. If the write operation of the existing data is completed later than the write operation of the real-time incremental data, the existing data will overwrite the new data already written in the target database, leading to inconsistencies between the source and target databases and failing to effectively guarantee the eventual consistency of the fully migrated data.

[0004] Therefore, how to improve the final consistency of the fully migrated data while ensuring zero downtime and zero impact on business operations is a problem that urgently needs to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a data migration method, device and storage medium, which aims to improve the final consistency of the fully migrated data while ensuring zero business downtime and zero impact.

[0006] To achieve the above objectives, this application provides a data migration method, the data migration method comprising: In response to a data migration command, the system obtains the first existing data in the source database and the initial identification information of the first existing data. If the initial identification information exists in the target database, obtain the first version information of the first existing data, and obtain the second version information of the data already written in the target database; When the data update sequence represented by the first version information is later than the data update sequence represented by the second version information, the first existing data is written to the target database.

[0007] In one embodiment, the method further includes: In response to a business write request, obtain the incremental data from the business write request; The first existing data and the incremental data are used as the full data in the source database; Based on the full data, the written data in the target database is verified to obtain a set of difference data between the target database and the source database; Based on the difference data set, perform data repair operations on the written data until the data differences between the target database and the source database meet the preset consistency conditions.

[0008] In one embodiment, the step of verifying the written data in the target database based on the full dataset to obtain a set of difference data between the target database and the source database includes: Based on the full data, a full data verification is performed on the written data to obtain the first difference data; Based on the full data, incremental data verification is performed on the written data to obtain the second difference data; The first difference data and the second difference data are grouped into the difference data set.

[0009] In one embodiment, the first difference data includes first missing data and first redundant data, and the step of performing full data verification on the written data to obtain the first difference data includes: The data in the full data excluding the data that has already been written is taken as the first missing data; The data in the written data other than the full data is taken as the first redundant data.

[0010] In one embodiment, the first difference data further includes first field difference data, and the step of performing full data verification on the written data based on the full data to obtain the first difference data includes: Obtain the first identifier information of each data in the full data, and obtain the second identifier information of each data in the written data; For any one of the first identifier information, the first identifier information is matched with each of the second identifier information; If the first identification information is successfully matched with the third identification information in each of the second identification information, the data corresponding to the first identification information in the full data is taken as the first source data, and the data corresponding to the third identification information in the written data is taken as the second target data. By comparing the first source data and the second target data, the difference between the second target data and the first source data is used as the difference data of the first field.

[0011] In one embodiment, the second difference data includes second missing data and second redundant data. The step of performing incremental data verification on the written data based on the full data to obtain the second difference data includes: Determine whether the source database has been updated within a preset historical time period; If the source database has updated data within a preset historical period, obtain the third target data corresponding to the updated data in the target database; The data in the updated data other than the data in the third target end is taken as the second missing data, and the data in the data in the third target end other than the updated data is taken as the second redundant data.

[0012] In one embodiment, the second difference data further includes second field difference data, and the step of performing incremental data verification on the written data based on the full data to obtain the second difference data includes: If the source database has updated data within a preset historical period, the fourth identification information of each data in the updated data is obtained, and the fifth identification information of each data in the third target data, excluding the second redundant data, is obtained. For any of the fourth identification information, the fourth identification information is matched with each of the fifth identification information; If the fourth identification information is successfully matched with the sixth identification information in each of the fifth identification information, the data corresponding to the fourth identification information in the updated data is used as the updated end data, and the data corresponding to the sixth identification information in the third target end data is used as the fourth target end data. By comparing the updated data and the fourth target data, the difference between the fourth target data and the updated data is used as the second field difference data.

[0013] In one embodiment, the step of performing data repair operation on the written data based on the difference data set includes: A data repair strategy is determined based on the discrepancy data set, wherein the data repair strategy includes at least one of data writing, data deletion, and data updating; Traverse the set of discrepancies and perform automatic repair operations on the written data based on the data repair strategy.

[0014] Furthermore, to achieve the above objectives, this application also provides a data migration apparatus, the data migration apparatus comprising: The response module is used to respond to the data migration instruction by obtaining the first existing data in the source database and the initial identification information of the first existing data. The acquisition module is used to acquire the first version information of the first existing data and the second version information of the data already written to the target database, provided that the initial identification information exists in the target database. The writing module is used to write the first existing data to the target database when the data update time sequence represented by the first version information is later than the data update time sequence represented by the second version information.

[0015] In addition, to achieve the above objectives, this application also proposes a data migration device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data migration method as described above.

[0016] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, on which a program implementing the data migration method is stored, and the program implementing the data migration method is executed by a processor to implement the steps of the data migration method as described above.

[0017] In addition, to achieve the above objectives, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data migration method described above.

[0018] This application provides a data migration method. In response to a data migration instruction, this application obtains first existing data and initial identification information of the first existing data in the source database; if the initial identification information exists in the target database, it obtains first version information of the first existing data and second version information of the data already written in the target database; when the data update time sequence represented by the first version information is later than the data update time sequence represented by the second version information, it writes the first existing data to the target database.

[0019] In summary, this application, upon receiving a data migration instruction, obtains the first existing data to be migrated from the source database and its identification information. If the identification information exists in the target database, it compares the first version information of the first existing data with the second version information of the data already written to the target database. If the data update sequence represented by the first version information is later than the data update sequence represented by the second version information, the first existing data is written to the target database. Thus, compared to traditional migration methods that use a dual-write mechanism that leads to new data being overwritten by old data, this application introduces a version-based timing judgment mechanism during the existing data writing stage. This ensures that the data in the target database is the newer version, thereby preventing older existing data from overwriting new data already written to the target database, and ultimately improving the final consistency of the data during a full data migration between the source and target databases. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the first embodiment of the data migration method of this application; Figure 2 This is a schematic diagram of the full data verification process involved in one embodiment of the data migration method of this application; Figure 3 This is a schematic diagram of the incremental data verification process involved in one embodiment of the data migration method of this application; Figure 4 This is a schematic diagram of the multi-mechanism collaborative architecture involved in an embodiment of the data migration method of this application; Figure 5 This is a schematic diagram of the module structure of the data migration device of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the data migration method in the embodiments of this application.

[0023] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] Currently, common MongoDB data migration methods typically send write requests to both the source and target databases simultaneously during the migration of existing data, aiming to achieve zero business downtime during the migration process. However, in complex scenarios where existing and incremental data are written in parallel, the order of completion of the two writes becomes uncertain when data with the same primary key is updated in real time during the existing data synchronization task. If the write operation of the existing data is completed later than the write operation of the real-time incremental data, the existing data will overwrite the new data already written in the target database, leading to inconsistencies between the source and target databases and failing to effectively guarantee the eventual consistency of the fully migrated data.

[0027] Therefore, how to improve the final consistency of the fully migrated data while ensuring zero downtime and zero impact on business operations is a problem that urgently needs to be solved.

[0028] The main solution of this application is: in response to a data migration instruction, obtain the first existing data and the initial identification information of the first existing data in the source database; if the initial identification information exists in the target database, obtain the first version information of the first existing data and obtain the second version information of the data already written in the target database; when the data update time sequence represented by the first version information is later than the data update time sequence represented by the second version information, write the first existing data to the target database.

[0029] This application, upon receiving a data migration instruction, obtains the first existing data to be migrated from the source database and its identification information. If the identification information exists in the target database, it compares the first version information of the first existing data with the second version information of the data already written to the target database. If the data update sequence represented by the first version information is later than the data update sequence represented by the second version information, the first existing data is written to the target database. Thus, compared to traditional migration methods that use a dual-write mechanism leading to new data being overwritten by old data, this application introduces a version-based timing judgment mechanism during the existing data writing stage. This ensures that the data in the target database is the newer version, thereby preventing older existing data from overwriting new data already written to the target database, and ultimately improving the final consistency of the data during a full data migration between the source and target databases.

[0030] It should be noted that the execution subject of the data migration method in various embodiments of this application can be a data migration device, or a computing service device with data processing, network communication, and program execution functions, such as a server, computer, tablet computer, personal computer, mobile phone, etc., or a data migration system capable of realizing the above functions, etc. This embodiment does not specifically limit it in this way. The following uses a data migration device as the execution subject as an example to describe this embodiment and the following embodiments.

[0031] Based on this, this application proposes a data migration method according to the first embodiment, please refer to... Figure 1 The data migration method includes steps S10 to S30: Step S10: In response to the data migration instruction, obtain the first existing data in the source database and the initial identification information of the first existing data.

[0032] A data migration command is a control signal that triggers the data migration device to perform a data migration operation. This command is triggered by the user through the migration management interface or the upper-level business system. The source database refers to the data storage system where the data to be migrated currently resides; that is, the source of the data migration. The initial existing data refers to historical data that existed in the source database before the data migration command was triggered; that is, the existing data to be migrated. Initial identification information refers to the primary key or unique key information used to uniquely identify each piece of data in the initial existing data, such as the primary key field value in MongoDB.

[0033] In response to the data migration command, the data migration device acquires the first existing data in the source database and simultaneously acquires the initial identification information of each data item in the first existing data, so as to perform matching and judgment based on the initial identification information in the target database later.

[0034] For example, in a MongoDB data migration scenario, the data migration device responds to the data migration command triggered by the user, obtains a full data snapshot of the source database at a specified time point through the MongoDB cursor mechanism, uses the snapshot data as the first existing data, and extracts the primary key field values ​​of each data in the snapshot as initial identification information.

[0035] In one feasible implementation, after the data migration device acquires the first existing data, it pushes the first existing data to the Kafka (Apache Kafka, distributed message queue) message queue for buffering and forwarding, using Kafka to achieve traffic shaping and asynchronous decoupling. During the data consumption stage, after data standardization processing such as field mapping, non-empty verification, length verification, format conversion, batch insertion and transaction control are performed by Java service, the data is then written to the target database, thereby avoiding performance overload and service jitter of the target database caused by the high throughput of writing full snapshots of the source database.

[0036] Step S20: If the initial identification information exists in the target database, obtain the first version information of the first existing data and obtain the second version information of the data already written in the target database.

[0037] The target database refers to the intended data storage system for the data migration, i.e., the target end of the data migration. First version information refers to the version identifier information carried by the data to be written in the first set of existing data, used to characterize the update sequence of that data. Second version information refers to the version identifier information carried by the data already written to the target database, used to characterize the update sequence of that written data. Version information is usually represented by a version number; a larger version number indicates a later data update sequence.

[0038] The data migration device matches the target database based on the initial identification information. If data consistent with the initial identification information exists in the target database, it obtains the first version information of the first existing data and the second version information of the data already written to the target database. If the initial identification information does not exist in the target database, there is no need to perform version comparison, and the first existing data can be directly written to the target database.

[0039] Step S30: When the data update sequence represented by the first version information is later than the data update sequence represented by the second version information, the first existing data is written to the target database.

[0040] Data update sequence refers to the order in which data is updated or written, usually represented by version numbers, with larger version numbers indicating later updates. Writing refers to the process of storing existing data in the target database according to its data structure requirements.

[0041] The data migration device compares the data update timelines represented by the first version information and the second version information. If the data update timeline represented by the first version information is later than that represented by the second version information, it indicates that the existing data is newer than the data already written in the target database. In this case, the first existing data is written to the target database. Otherwise, it indicates that the data already written in the target database is newer than the existing data. In this case, no write operation is performed on the existing data to avoid the existing data overwriting the newer data already written in the target database.

[0042] For example, if the version number of a certain data in the existing data is 3, and the version number of the corresponding data in the target database is 2, the data migration device writes the existing data to the target database because the update sequence represented by version number 3 is later than that of version number 2; if the version number of the corresponding data in the target database is 5, the data migration device does not perform the write operation.

[0043] This embodiment of the application, upon receiving a data migration instruction, obtains the first existing data to be migrated from the source database and its identification information. If the identification information exists in the target database, it compares the first version information of the first existing data with the second version information of the data already written to the target database. If the data update sequence represented by the first version information is later than the data update sequence represented by the second version information, the first existing data is written to the target database. Thus, compared to traditional migration methods that use a dual-write mechanism that causes new data to be overwritten by old data, this embodiment introduces a version-based timing judgment mechanism during the existing data writing stage. This ensures that the data in the target database is the newer version, thereby preventing older existing data from overwriting new data already written to the target database, and improving the eventual consistency of the full data migration between the source and target databases.

[0044] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, the data migration method further includes a data verification and automatic repair mechanism, and the data migration method further includes steps S40~S70: Step S40: In response to the business write request, obtain the incremental data in the business write request.

[0045] A business write request refers to a data write request initiated by a business system to the source database during data migration, such as a request to add, modify, or delete data. Incremental data refers to new business data generated during the data migration process that is not yet included in the initial existing data.

[0046] In response to a business write request, the data migration device obtains the incremental data in the business write request and uses the incremental data as supplementary data in the data migration process, together with the existing data, for subsequent data verification and repair.

[0047] This application does not limit the triggering method of business write requests. For example, in one feasible implementation, the data migration device obtains incremental data in the business write request by listening to the change data capture log of the source database; in another feasible implementation, the data migration device obtains incremental data in the business write request by intercepting the data access layer request between the business system and the source database.

[0048] In the first feasible implementation, the data migration device subscribes to the oplog (operation log) of the source database through the CDC (Change Data Capture) component. When the business system initiates a write operation to the source database, the CDC component captures the change event corresponding to the operation in real time and pushes the incremental data in the change event to the data migration device.

[0049] In a second feasible implementation, the data migration device implants an interceptor in the data access layer between the business system and the source database. When the business system initiates a data write request, the interceptor synchronously obtains the write data in the request before the request reaches the source database, and reports the write data as incremental data to the data migration device.

[0050] Step S50: The first existing data and the incremental data are used as the full data in the source database.

[0051] Full data refers to a complete set of data that consists of both existing data and incremental data, covering all data records in the source database.

[0052] The data migration device merges the acquired existing data with the incremental data, and uses the merged data set as the full data in the source database, serving as the benchmark data for subsequent data verification.

[0053] Step S60: Verify the written data in the target database based on the full data to obtain a set of difference data between the target database and the source database.

[0054] Validation refers to the process of comparing and analyzing the data already written to the target database against the full dataset to identify data discrepancies between the target database and the source database. The discrepancy data set refers to the collection of data in the target database that differs from the source database, as determined through validation. This discrepancy data includes missing data, redundant data, and data with different fields.

[0055] The data migration device uses the full dataset as a benchmark to verify the data already written to the target database. It then categorizes the differences between the target database and the source database into a difference data set for subsequent data repair.

[0056] In this embodiment, step S60 may include: Step S601: Perform full data verification on the written data based on the full data to obtain the first difference data; Step S602: Perform incremental data verification on the written data based on the full data to obtain the second difference data; Step S603: The first difference data and the second difference data are included in the difference data set.

[0057] Full data verification refers to the process of comparing all written data in the target database with the full dataset as a baseline to identify missing, redundant, and field-difference data. Incremental data verification refers to the process of comparing the updated data in the source database with the corresponding data in the target database within a preset historical time period to identify missing, redundant, and field-difference data in the incremental dimension.

[0058] The data migration device first performs a full data verification on the data already written to the target database based on the full data, obtaining the difference data between the target database and the source database in the full data dimension (hereinafter referred to as the first difference data for distinction); then it performs incremental data verification on the data already written based on the full data, obtaining the difference data between the target database and the updated data set in the updated data dimension (hereinafter referred to as the second difference data for distinction); finally, the first difference data and the second difference data are merged into the difference data set, so that the difference data set simultaneously covers the data differences generated in both stock migration and real-time dual-write scenarios.

[0059] In this embodiment, the first difference data includes first missing data and first redundant data, and step S601 may include: Step S6011: Take the data in the full data other than the data that has been written as the first missing data; Step S6012: The data in the written data other than the full data is taken as the first redundant data.

[0060] The first missing data refers to data that exists in the full dataset but has not been written to the target database; that is, data that is missing from the target database relative to the source database. The first redundant data refers to data that has been written to the target database but is not part of the full dataset; that is, data that is redundant from the source database to the target database.

[0061] The data migration device compares the full data with the data already written in the target database. Data in the full data other than the data already written is identified as the first missing data, and data in the written data other than the full data is identified as the first redundant data, thus achieving full identification of missing and redundant data.

[0062] For example, if the full dataset contains 1000 records and the target database has written 998 records, then the two extra records in the full dataset will be considered as the first missing data; if the target database has actually written 1002 records, then the two extra records will be considered as the first redundant data.

[0063] In this embodiment, the first difference data further includes first field difference data, and step S601 may further include: Step S6013: Obtain the first identification information of each data in the full data, and obtain the second identification information of each data in the written data; Step S6014: For any one of the first identification information, match the first identification information with each of the second identification information respectively; Step S6015: If the first identification information is successfully matched with the third identification information in each of the second identification information, the data corresponding to the first identification information in the full data is taken as the first source data, and the data corresponding to the third identification information in the written data is taken as the second target data. Step S6016: Compare the first source data and the second target data, and use the difference data of the second target data relative to the first source data as the first field difference data.

[0064] The first identifier refers to the unique identifier of each data entry in the full dataset. The second identifier refers to the unique identifier of each data entry in the written dataset. The first source data refers to the data in the full dataset that successfully matches the identifier of the written data. The second target data refers to the data in the written dataset that successfully matches the identifier of the full dataset. The first field difference data refers to data in the two databases where the identifiers match but the field values ​​are inconsistent.

[0065] The data migration device acquires the first identification information of each data in the full data and the second identification information of each data in the written data. It matches each first identification information with each second identification information. When a match is successful, the corresponding data in the full data is taken as the first source data and the corresponding data in the written data is taken as the second target data. The device compares the first source data and the second target data field by field and takes the difference data between the second target data and the first source data as the first field difference data.

[0066] For example, the identifier of order record A in the full data is 001, and the identifier of record B written to the target database is also 001. The two match successfully. The comparison finds that the order status field of record B is inconsistent with that of record A. Therefore, the field difference of record B relative to record A is taken as the first field difference data.

[0067] For example, such as Figure 2The diagram illustrates the full data verification process. First, the first identifier information of each data point in the full data and the second identifier information of each data point in the written data are obtained. Next, each first identifier information is matched with each second identifier information. When a match is successful, the corresponding data in the full data is taken as the first source data, and the corresponding data in the written data is taken as the second target data. The first source data and the second target data are compared field by field, and the difference between the second target data and the first source data is taken as the first field difference data. At the same time, the data in the full data other than the written data is taken as the first missing data, and the data in the written data other than the full data is taken as the first redundant data.

[0068] In this embodiment, the second difference data includes second missing data and second redundant data. Step S602 may include: Step S6021: Determine whether the source database has updated data within a preset historical time period; Step S6022: If the source database has updated data within a preset historical time period, obtain the third target data corresponding to the updated data in the target database. Step S6023: The data in the updated data other than the data in the third target end is taken as the second missing data, and the data in the data in the third target end other than the updated data is taken as the second redundant data.

[0069] The preset historical time period refers to the pre-configured time scan range used for incremental data verification, such as the previous day (T-1 day). Updated data refers to data newly generated or updated in the source database within the preset historical time period. Third target data refers to new data written to the target database through dual writes. Second missing data refers to data in the updated data that exists in the source database but has not been written to the target database. Second redundant data refers to data that has been written to the target database but does not exist in the updated data of the source database.

[0070] The data migration device determines whether the source database has updated data within a preset historical period. If there is updated data, it obtains the newly written data in the target database through the dual-write mechanism as the third target data, takes the data in the updated data other than the third target data as the second missing data, and takes the data in the third target data other than the updated data as the second redundant data.

[0071] For example, if the preset historical period is the previous day, and the source database updated 5 records the previous day, of which 4 records have been synchronously written to the target database, then the 1 record that has not been written will be regarded as the second missing data; if there are 3 records in the target database that are inconsistent with the updated data identifier, then these 3 records will be regarded as the second redundant data.

[0072] In this embodiment, the second difference data further includes second field difference data, and step S602 may further include: Step S6024: If the source database has updated data within a preset historical time period, obtain the fourth identification information of each data in the updated data, and obtain the fifth identification information of each data in the third target data except for the second redundant data. Step S6025: For any of the fourth identification information, match the fourth identification information with each of the fifth identification information; Step S6026: If the fourth identification information is successfully matched with the sixth identification information in each of the fifth identification information, the data corresponding to the fourth identification information in the updated data is used as the updated end data, and the data corresponding to the sixth identification information in the third target end data is used as the fourth target end data. Step S6027: Compare the updated data and the fourth target data, and use the difference data of the fourth target data relative to the updated data as the second field difference data.

[0073] The fourth identifier refers to the unique identifier of each data entry in the updated data. The fifth identifier refers to the unique identifier of each data entry in the third target data. The updated data refers to the data in the updated data that successfully matches the identifier of the third target data. The fourth target data refers to the data in the third target data that successfully matches the identifier of the updated data. The second field difference data refers to data in the updated data and the corresponding data in the target database where the identifier matches but the field values ​​are inconsistent.

[0074] The data migration device acquires the fourth identification information of each data in the updated data and the fifth identification information of each data in the third target data. It matches each fourth identification information with each fifth identification information. When a match is successful, the corresponding data in the updated data is used as the updated data and the corresponding data in the third target data is used as the fourth target data. The updated data and the fourth target data are compared field by field, and the difference between the fourth target data and the updated data is used as the second field difference data.

[0075] For example, the identifier of record C in the source database is 005, and the identifier of record D in the target database is also 005. The two are matched successfully. The comparison finds that the balance field of record D is inconsistent with that of record C. The field difference between record D and record C is then used as the second field difference data.

[0076] For example, such as Figure 3The diagram illustrates the incremental data verification process. First, it determines whether there is updated data in the source database within a preset historical period. If so, it determines whether the updated data exists in the target database. If updated data exists in the target database, it obtains the third target data corresponding to the fourth identifier information of the updated data in the target database. The data in the updated data other than the third target data is taken as the second missing data, and the data in the third target data other than the updated data is taken as the second redundant data. Simultaneously, it obtains the fourth identifier information of each data in the updated data and the fifth identifier information of each data in the third target data. It matches each fourth identifier information with each fifth identifier information. When a match is successful, the corresponding data in the updated data is taken as the updated data, and the corresponding data in the third target data is taken as the fourth target data. The updated data and the fourth target data are compared field by field, and the difference between the fourth target data and the updated data is taken as the second field difference data.

[0077] Step S70: Perform a data repair operation on the written data based on the difference data set until the data difference between the target database and the source database meets the preset consistency conditions.

[0078] Data repair operation refers to the correction process performed on the written data in the target database according to a preset data repair strategy, targeting each discrepancy data in the discrepancy data set, to eliminate data differences between the target database and the source database. The preset consistency conditions refer to the judgment conditions corresponding to when the data differences between the target database and the source database meet preset requirements, such as the discrepancy data set being empty or the number of discrepancy data being less than a preset threshold.

[0079] In this embodiment, step S70 may include: Step S701: Determine a data repair strategy based on the discrepancy data set, wherein the data repair strategy includes at least one of data writing, data deletion, and data updating; Step S702: Traverse the set of differential data and perform automatic repair operations on the written data based on the data repair strategy.

[0080] A data repair strategy refers to a set of pre-configured or dynamically determined processing rules for various types of discrepancies in a dataset. These rules specify the specific repair operations to be performed on different types of discrepancies. Specifically, the data writing strategy is used to add data present in the source database but missing in the target database to the target database; the data deletion strategy is used to remove redundant data present in the target database but absent in the source database; and the data update strategy is used to correct data with inconsistent field values ​​between the target and source databases to the values ​​found in the source database. The determination of the repair strategy is based on the specific type of discrepancy data and the pre-defined consistency assurance requirements.

[0081] Specifically, the data migration device traverses the discrepancy data set, identifies the discrepancy type of each discrepancy data entry, and matches the corresponding data repair strategy based on the discrepancy type: if the discrepancy data is the first or second missing data, a data write operation is performed to write the corresponding data record from the source database into the target database; if the discrepancy data is the first or second redundant data, a data deletion operation is performed to remove the redundant data record from the target database; if the discrepancy data is the first or second field discrepancy data, a data update operation is performed to correct the value of the corresponding field in the target database to the value of that field in the source database. In cases where multiple discrepancy types exist simultaneously, the data migration device performs repair operations sequentially according to the priority order of data write, data update, and data deletion. After each round of repair is completed, a full data verification and incremental data verification are re-executed. If the discrepancy data set is not cleared, the next round of repair continues until the discrepancy data set is empty.

[0082] For example, in a MongoDB data migration task, a full verification revealed that the target database was missing two historical order records and contained one redundant record. Field comparison also revealed an inconsistency in the status field of one order record. Incremental verification found that one of the five records updated that day had not been written to the target database. The data migration device traversed the discrepancy data set. For the three missing records (two historical and one incremental), a data writing strategy was executed to add the corresponding record from the source database to the target database. For the redundant record, a data deletion strategy was executed to remove it from the target database. For the record with a field difference, a data update strategy was executed to correct the status field of the record in the target database to the value from the source database. After the repair was completed, a re-verification confirmed that the discrepancy data set was empty, and the repair process ended.

[0083] This embodiment of the application obtains incremental data in response to business write requests, merges the first existing data and the incremental data into full data, performs full and incremental verification on the data already written to the target database to obtain a set of differing data, and performs automatic repair operations on the differing data based on preset repair parameters. Thus, compared to traditional solutions that only complete data synchronization and migration, this embodiment of the application, through the above-mentioned intelligent data comparison and automatic repair mechanism, can promptly detect and repair abnormal data such as data loss, data redundancy, and field differences generated during existing data migration, real-time dual writing, and traffic switching, improving the consistency and accuracy of data migration.

[0084] Based on the first or second embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to the first or second embodiment described above can be referred to the above description and will not be repeated hereafter. To more completely illustrate the technical solution of this application, it should be noted that the data migration scheme of this application consists of multiple mutually cooperating processing mechanisms, specifically including a stock data synchronization mechanism, a real-time traffic dual-write mechanism, a data verification and repair mechanism, and a traffic switch switching mechanism. Among them, the stock data synchronization mechanism is used to obtain the stock data in the source database under the trigger of the data migration command, and synchronize it to the target database after data standardization processing, so as to avoid the stock data overwriting the newer real-time data in the target database; the real-time traffic dual-write mechanism is used to make the business write request write to the source database and the target database simultaneously during the data migration to ensure uninterrupted business flow; the data verification and repair mechanism is used to intelligently compare the stock data and the real-time data, and promptly detect and automatically repair data differences; the traffic switch switching mechanism is used to flexibly control the read and write switch through the parameter center to achieve smooth flow switching and fast rollback.

[0085] Furthermore, the specific implementation steps of the existing data synchronization mechanism have been detailed in Embodiment 1 above, and the specific implementation steps of the data verification and repair mechanism have been detailed in Embodiment 2 above. This embodiment focuses on supplementing the explanation of the real-time traffic dual-write mechanism and the traffic switch switching mechanism. The following describes the real-time traffic dual-write mechanism and the traffic switch switching mechanism in conjunction with specific steps, which include steps A10 to A30: Step A10: Before the existing data extraction stage, configure the data migration device to run with dual write to the source database and target database and read from the source database through the parameter center. This will enable all business write requests to be written to both the source database and the target database at the same time, while read requests will still be routed to the source database.

[0086] The parameter center is a control center used to centrally configure the operating parameters of data migration equipment. Operations and maintenance personnel can dynamically adjust configuration items such as dual-write status, read / write switches, and traffic switching rules through the parameter center. Real-time dual-write means that during data migration, all business write requests are simultaneously written to both the source and target databases, ensuring that the target database maintains incremental data consistent with the source database in real time, while read requests are still routed to the source database to guarantee business continuity.

[0087] Before the data migration equipment extracts existing data, it configures its operating status to dual-write source database and target database, and read source database through the parameter center. Subsequently, each write request initiated by the business system will write to both the source database and the target database simultaneously, while read requests will still access the source database. This ensures uninterrupted business operation while enabling the target database to continuously receive real-time incremental data, laying the foundation for smooth data switching in the future.

[0088] For example, before the migration of existing data is started, the operation and maintenance personnel check the "dual write source database and target database, read source database" option in the configuration interface of the parameter center and save it. The parameter center will then send the configuration to the data migration device, which will then open the dual write channel. After that, business write requests will simultaneously fall into the source database and the target database.

[0089] Step A20: During the data migration and verification process, the daily new abnormal data is monitored in real time through the parameter center. If no new abnormal data is added, the traffic is gradually switched to the target database according to the preset traffic switching rules.

[0090] Anomaly data monitoring refers to a monitoring component used to store and manage inconsistent data. It primarily records two types of data differences: one is data that exists in the source database but not in the target database, and the other is data that exists in both the source and target databases but differs in a certain field. It records the specific field name and corresponding value of the inconsistency in detail. Traffic switching rules refer to the switching strategy configured in the parameter center to gradually migrate business read traffic from the source database to the target database, such as switching by percentage level or by business module in batches.

[0091] During the data migration and verification process, the data migration device observes the daily new abnormal data in real time through the parameter center. If there is no new abnormal data within the preset observation period, it indicates that the dual-write status is normal and the data consistency is good. Then, according to the preset traffic switching rules, the read traffic is gradually switched to the target database. During the switching process, the changes in abnormal data are continuously observed until all traffic is routed to the target database.

[0092] For example, if the operation and maintenance personnel observe abnormal data monitoring through the parameter center and there is no new abnormal data for three consecutive days, then the business read traffic will be gradually switched to the target database according to the percentage switching rule of "10%, 30%, 60%, 100%". After each level of switching, the system will be observed for a certain period of time, and the next level of switching will be carried out only after confirming that there are no abnormalities.

[0093] Step A30: In the event of an anomaly in the target database, the route is switched back to the source database through the parameter center to achieve a fast rollback.

[0094] Rollback refers to the process of switching the business route back to the source database through the parameter center when an anomaly occurs in the target database after a traffic switch, so as to restore the business to the running state before the migration and ensure business continuity and data security.

[0095] After traffic is switched to the target database, if the data migration device detects data anomalies in the target database through abnormal data monitoring or discovers problems such as link backlog or write failures through monitoring, it will immediately switch the route back to the source database through the parameter center to achieve a rapid rollback. Since business data is always written twice during the migration, the data in the target database and the source database remain consistent, and the business is unaware of the rollback process.

[0096] For example, after all traffic has been switched to the target database, abnormal data monitoring shows that new abnormal data has appeared in the target database and continues to grow. The operation and maintenance personnel can switch the route back to the source database with one click through the parameter center. The business immediately resumes the read and write state of the source database. Then, the abnormal data is investigated and repaired. After the abnormality is eliminated, the traffic switch is performed again.

[0097] For example, such as Figure 4 The diagram illustrates a multi-mechanism collaborative architecture. Under the unified management of the parameter center, it shows the overall architectural relationship of the existing data synchronization mechanism, real-time traffic dual-write mechanism, data verification mechanism, automatic repair mechanism, and traffic switch switching mechanism surrounding the data migration operation between the source and target databases. Specifically, the existing data synchronization mechanism is responsible for writing the existing data in the source database to the target database after version verification; the real-time traffic dual-write mechanism ensures that business write requests during migration simultaneously fall into both the source and target databases; the data verification and automatic repair mechanisms intelligently compare full and incremental data and automatically eliminate data differences; and the traffic switch switching mechanism dynamically controls read and write routes through the parameter center, achieving smooth traffic switching and rapid rollback.

[0098] This application embodiment achieves dynamic management and control of dual-write configuration, real-time observation of abnormal data, and smooth switching and rapid rollback of traffic through a parameter center. This enables the four stages of existing data synchronization, real-time dual-write, data verification and repair, and traffic switching to work in synergy, forming a complete migration closed loop with self-discovery and rapid self-healing capabilities. Thus, compared to traditional solutions that only focus on data synchronization and migration, this application embodiment, while ensuring zero downtime and zero-awareness of business operations, effectively addresses various abnormal scenarios throughout the migration process through the coordinated operation of the aforementioned multiple mechanisms, significantly improving the consistency, accuracy, and operational reliability of data migration.

[0099] This application also provides a data migration apparatus, please refer to... Figure 5 The data migration device includes: Response module 10 is used to respond to a data migration instruction by obtaining the first existing data in the source database and the initial identification information of the first existing data. The acquisition module 20 is used to acquire the first version information of the first existing data and the second version information of the data already written in the target database when the initial identification information exists in the target database. The writing module 30 is used to write the first existing data to the target database when the data update time sequence represented by the first version information is later than the data update time sequence represented by the second version information.

[0100] Optionally, the data migration apparatus further includes a repair module, the repair module being used for: In response to a business write request, obtain the incremental data from the business write request; The first existing data and the incremental data are used as the full data in the source database; Based on the full data, the written data in the target database is verified to obtain a set of difference data between the target database and the source database; Based on the difference data set, perform data repair operations on the written data until the data differences between the target database and the source database meet the preset consistency conditions.

[0101] Optionally, the repair module is further configured to: Based on the full data, a full data verification is performed on the written data to obtain the first difference data; Based on the full data, incremental data verification is performed on the written data to obtain the second difference data; The first difference data and the second difference data are grouped into the difference data set.

[0102] Optionally, the repair module is further configured to: The first difference data includes first missing data and first redundant data; The data in the full data excluding the data that has already been written is taken as the first missing data; The data in the written data other than the full data is taken as the first redundant data.

[0103] Optionally, the repair module is further configured to: The first difference data also includes the first field difference data; Obtain the first identification information of each data in the full data, and obtain the second identification information of each data in the written data excluding the first redundant data; For any one of the first identifier information, the first identifier information is matched with each of the second identifier information; If the first identification information is successfully matched with the third identification information in each of the second identification information, the data corresponding to the first identification information in the full data is taken as the first source data, and the data corresponding to the third identification information in the written data is taken as the second target data. By comparing the first source data and the second target data, the difference between the second target data and the first source data is used as the difference data of the first field.

[0104] Optionally, the repair module is further configured to: The second difference data includes second missing data and second redundant data; Determine whether the source database has been updated within a preset historical time period; If the source database has updated data within a preset historical period, obtain the third target data corresponding to the updated data in the target database; The data in the updated data other than the data in the third target end is taken as the second missing data, and the data in the data in the third target end other than the updated data is taken as the second redundant data.

[0105] Optionally, the repair module is further configured to: The second difference data also includes the second field difference data; If the source database has updated data within a preset historical period, the fourth identification information of each data in the updated data is obtained, and the fifth identification information of each data in the third target data, excluding the second redundant data, is obtained. For any of the fourth identification information, the fourth identification information is matched with each of the fifth identification information; If the fourth identification information is successfully matched with the sixth identification information in each of the fifth identification information, the data corresponding to the fourth identification information in the updated data is used as the updated end data, and the data corresponding to the sixth identification information in the third target end data is used as the fourth target end data. By comparing the updated data and the fourth target data, the difference between the fourth target data and the updated data is used as the second field difference data.

[0106] Optionally, the repair module is further configured to: A data repair strategy is determined based on the discrepancy data set, wherein the data repair strategy includes at least one of data writing, data deletion, and data updating; Traverse the set of discrepancies and perform automatic repair operations on the written data based on the data repair strategy.

[0107] The data migration apparatus provided in this application, employing the data migration method described in the above embodiments, can solve the technical problem in existing data migration schemes where existing data overwrites new data already written to the target database, leading to inconsistencies between the source and target databases. Compared to the prior art, the beneficial effects of the data migration apparatus provided in this application are the same as those of the data migration method described in the above embodiments, and other technical features in the data migration apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0108] This application provides a data migration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the data migration method in Embodiment 1 above.

[0109] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a data migration device suitable for implementing the embodiments of this application. Figure 6 The data migration device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0110] like Figure 6 As shown, the data migration device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the data migration device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus 1005.

[0111] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the data migration method of the embodiments of this application.

[0112] The data migration device provided in this application, employing the data migration method described in the above embodiments, can solve the technical problem in existing data migration schemes where existing data overwrites new data already written to the target database, leading to inconsistencies between the source and target databases. Compared with the prior art, the beneficial effects of the data migration device provided in this application are the same as those of the data migration method provided in the above embodiments, and other technical features of this data migration device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0113] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0115] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data migration method described in the above embodiments.

[0116] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0117] The aforementioned computer-readable storage medium may be included in the data migration device; or it may exist independently and not be assembled into the data migration device.

[0118] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the data migration device, the data migration device: in response to a data migration instruction, acquires first existing data and initial identification information of the first existing data in the source database; if the initial identification information exists in the target database, acquires first version information of the first existing data and acquires second version information of data already written to the target database; and if the data update time sequence represented by the first version information is later than the data update time sequence represented by the second version information, writes the first existing data to the target database.

[0119] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0121] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0122] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data migration method. This solves the technical problem that existing data migration schemes lack the ability to identify, correct, and automatically repair abnormal data, making it difficult to guarantee the consistency and accuracy of data migration. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data migration method provided in the above embodiments, and will not be repeated here.

[0123] An embodiment of the application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data migration method described above.

[0124] The computer program product provided in this application can solve the technical problem that existing data migration schemes cause existing data to overwrite new data written to the target database, resulting in inconsistencies between the source and target databases. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data migration method provided in the above embodiments, and will not be repeated here.

[0125] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A data migration method, characterized by, The data migration method includes: In response to a data migration command, the system obtains the first existing data in the source database and the initial identification information of the first existing data. If the initial identification information exists in the target database, obtain the first version information of the first existing data, and obtain the second version information of the data already written in the target database; When the data update sequence represented by the first version information is later than the data update sequence represented by the second version information, the first existing data is written to the target database.

2. The data migration method of claim 1, wherein, The method further includes: In response to a business write request, obtain the incremental data from the business write request; The first existing data and the incremental data are used as the full data in the source database; Based on the full data, the written data in the target database is verified to obtain a set of difference data between the target database and the source database; Based on the difference data set, perform data repair operations on the written data until the data differences between the target database and the source database meet the preset consistency conditions.

3. The data migration method as described in claim 2, characterized in that, The step of verifying the written data in the target database based on the full data to obtain the set of difference data between the target database and the source database includes: Based on the full data, a full data verification is performed on the written data to obtain the first difference data; Based on the full data, incremental data verification is performed on the written data to obtain the second difference data; The first difference data and the second difference data are grouped into the difference data set.

4. The data migration method as described in claim 3, characterized in that, The first difference data includes first missing data and first redundant data. The step of performing full data verification on the written data to obtain the first difference data includes: The data in the full data excluding the data that has already been written is taken as the first missing data; The data in the written data other than the full data is taken as the first redundant data.

5. The data migration method as described in claim 4, characterized in that, The first difference data also includes first field difference data. The step of performing full data verification on the written data based on the full data to obtain the first difference data includes: Obtain the first identification information of each data in the full data, and obtain the second identification information of each data in the written data excluding the first redundant data; For any one of the first identifier information, the first identifier information is matched with each of the second identifier information; If the first identification information is successfully matched with the third identification information in each of the second identification information, the data corresponding to the first identification information in the full data is taken as the first source data, and the data corresponding to the third identification information in the written data is taken as the second target data. By comparing the first source data and the second target data, the difference between the second target data and the first source data is used as the difference data of the first field.

6. The data migration method as described in claim 3, characterized in that, The second difference data includes second missing data and second redundant data. The step of performing incremental data verification on the written data based on the full data to obtain the second difference data includes: Determine whether the source database has been updated within a preset historical time period; If the source database has updated data within a preset historical period, obtain the third target data corresponding to the updated data in the target database; The data in the updated data other than the data in the third target end is taken as the second missing data, and the data in the data in the third target end other than the updated data is taken as the second redundant data.

7. The data migration method as described in claim 6, characterized in that, The second difference data also includes second field difference data. The step of performing incremental data verification on the written data based on the full data to obtain the second difference data includes: If the source database has updated data within a preset historical period, the fourth identification information of each data in the updated data is obtained, and the fifth identification information of each data in the third target data, excluding the second redundant data, is obtained. For any of the fourth identification information, the fourth identification information is matched with each of the fifth identification information; If the fourth identification information is successfully matched with the sixth identification information in each of the fifth identification information, the data corresponding to the fourth identification information in the updated data is used as the updated end data, and the data corresponding to the sixth identification information in the third target end data is used as the fourth target end data. By comparing the updated data and the fourth target data, the difference between the fourth target data and the updated data is used as the second field difference data.

8. The data migration method as described in claim 2, characterized in that, The step of performing data repair operation on the written data based on the difference data set includes: A data repair strategy is determined based on the discrepancy data set, wherein the data repair strategy includes at least one of data writing, data deletion, and data updating; Traverse the set of discrepancies and perform automatic repair operations on the written data based on the data repair strategy.

9. A data migration device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data migration method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data migration method as described in any one of claims 1 to 8.

Images