Data migration method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data migration method, device and equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that in response to a first instruction, first incremental data generated by a data generation device is written into a source system and a target system; if the target system is verified to meet the read-write condition, stock data in a source system and all first incremental data are synchronized to the target system, and the stock data are target data written into the source system before the first instruction generation moment; the first incremental data is target data written into a source system between the first instruction generation moment and the stock data synchronization starting moment. By adopting the technical scheme of the embodiment of the invention, the technical effect of improving the data migration speed can be achieved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data processing, and in particular, to a data migration method, apparatus, device, and storage medium. Background Art

[0002] For Internet applications, with the development of business, the old system usually has difficulty in supporting the application. To ensure the service quality of the application, it is necessary to deploy the business data in the old system to the new system through data migration to improve the business response ability.

[0003] Existing data migration methods usually first export data from the source system to the target file, and then import the target file into the target system.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] The data migration speed is slow, making it impossible for users to complete system switching in a short time. Summary of the Invention

[0006] Embodiments of the present invention provide a data migration method to solve the problem of slow data migration speed in existing data migration methods.

[0007] In a first aspect, embodiments of the present invention provide a data migration method, the method including:

[0008] In response to a first instruction, write first incremental data generated by a data generation device into the source system and the target system;

[0009] If the target system is verified to meet the read and write conditions, synchronize the stock data in the source system and all the first incremental data to the target system, where the stock data is the target data written into the source system before the generation time of the first instruction, and the first incremental data is the target data written into the source system between the generation time of the first instruction and the start time of stock data synchronization.

[0010] In a second aspect, embodiments of the present invention further provide a data migration apparatus, the apparatus including:

[0011] A dual-write module, configured to write first incremental data generated by a data generation device into the source system and the target system in response to a first instruction;

[0012] A data migration module, configured to synchronize the stock data and all first incremental data in the source system to the target system if the target system is verified to meet the read / write conditions, where the stock data is the target data written to the source system before the generation time of the first instruction, and the first incremental data is the target data written to the source system between the generation time of the first instruction and the start time of synchronizing the stock data.

[0013] In a third aspect, an embodiment of the present invention provides a server device, which includes:

[0014] One or more processors;

[0015] A memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data migration method provided in any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the data migration method provided in any embodiment of the present invention.

[0018] The embodiments in the above invention have the following advantages or beneficial effects:

[0019] In the technical solution of the data migration method provided in the embodiment of the present invention, from the generation time of the first instruction to the start of data migration, the first incremental data is written to the source system and the target system through a dual-write mode; if the target system is verified to meet the read / write conditions, it is defaulted that the read / write performance of the target system is good during data migration, that is, the second incremental data generated by the data generation device during data migration can be accurately written to the target system, and the second incremental data written to the target system is readable. Therefore, only the stock data and the first incremental data generated before data migration need to be synchronized to the target system; compared with the synchronization difficulty of real-time data, the synchronization difficulty of non-real-time data is lower and the synchronization speed is also faster. Therefore, the first incremental data and the stock data generated before data migration can be quickly synchronized from the source system to the target system; compared with first synchronizing the data to be migrated from the source system to the target file and then synchronizing the data to be migrated from the target file to the target system, directly synchronizing the stock data and the first incremental data from the source system to the target system further improves the data migration speed. Description of the Drawings

[0020] Figure 1 It is a schematic flowchart of the data migration method provided in the embodiment of the present invention;

[0021] Figure 2It is another flowchart of the data migration method provided by the embodiment of the present invention;

[0022] Figure 3 It is another flowchart of the data migration method provided by the embodiment of the present invention;

[0023] Figure 4 It is another flowchart of the data migration method provided by the embodiment of the present invention;

[0024] Figure 5 It is another flowchart of the data migration method provided by the embodiment of the present invention;

[0025] Figure 6A It is a schematic structural diagram of the data migration device provided by the embodiment of the present invention;

[0026] Figure 6B It is another schematic structural diagram of the data migration device provided by the embodiment of the present invention;

[0027] Figure 6C It is another schematic structural diagram of the data migration device provided by the embodiment of the present invention;

[0028] Figure 7 It is a schematic structural diagram of the server device provided by the embodiment of the present invention. Detailed implementation manners

[0029] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. In addition, it should be noted that for the convenience of description, only parts related to the present invention are shown in the drawings, rather than all the structures.

[0030] It should be noted that in the technical solutions of the present disclosure, in terms of the collection, gathering, updating, analysis, processing, use, transmission, storage, etc. of user personal information, they all comply with the provisions of relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken for user personal information to prevent illegal access to user personal information data, and to maintain the security of user personal information, network security, and national security.

[0031] Figure 1 It is a flowchart of the data migration method provided by the embodiment of the present invention. This embodiment is applicable to the situation of online data migration. This method can be executed by a data migration device integrated in a server device, and this device can be implemented in a software and / or hardware manner. As Figure 1 shown, this method specifically includes the following steps:

[0032] S110. In response to the first instruction, write the first incremental data generated by the data generation device into the source system and the target system.

[0033] In response to the first instruction, the dual-write mode is enabled, and the first incremental data generated by the data generation device is written to both the source system and the target system. Specifically, the target data generated by the data generation device after the generation time of the first instruction is used as the first incremental data, such as a settlement certificate file, and is written to both the source system and the target system at the same time.

[0034] In one embodiment, the source system is a financial service system based on Hbace, and the target system is an object storage system, and both are distributed storage systems.

[0035] S120. If the target system is verified to meet the read / write conditions, synchronize the stock data in the source system and all the first incremental data to the target system. The stock data is the target data written to the source system before the generation time of the first instruction, and the first incremental data is the target data written to the source system between the generation time of the first instruction and the start time of the stock data synchronization.

[0036] Among them, the stock data is also called historical data, and its target state formation time is before the generation time of the first instruction. Among them, the target state refers to the last modification state or generation state, and the latter is applicable to non-editable data.

[0037] In this embodiment, the data written to both the source system and the target system at the same time is used as incremental data. Specifically, the target data written to both the source system and the target system between the generation time of the first instruction and the start time of the stock data synchronization is used as the first incremental data; after the stock data starts to migrate, the incremental data written to both the source system and the target system is used as the second incremental data.

[0038] Among them, meeting the read / write conditions means that both the data write accuracy rate and the data read accuracy rate reach the corresponding threshold ranges.

[0039] Since the source system is stable and reliable, even if the writing of the first incremental data to the target system fails, there is the source system to back it up. Therefore, through the dual-write operation, the read / write performance of the target system can be verified while ensuring the security of data writing. For this reason, in this embodiment, after detecting the dual-write opening set duration, a read / write verification operation is performed on the target system to determine whether it meets the read / write conditions.

[0040] In one embodiment, existing read / write test cases are used to test whether the target system meets the read / write conditions.

[0041] If the target system is verified to meet the read and write conditions, it indicates that the target system is stable and accurate in terms of data writing and data querying, and the data written to the target system is accurate, that is, the data written to the source system is the same as the data written to the target system. Therefore, there is no need to synchronize the second incremental data written to the source system to the target system, and only the stock data and all the first incremental data in the source system need to be synchronized to the target system. Since the first incremental data is formed before the migration of the stock data, it is not data that changes in real time. Compared with the migration difficulty and speed of real-time generated data, its migration difficulty is lower and its migration speed is faster. Therefore, the overall migration speed of the stock data and the first incremental data in this embodiment is faster.

[0042] In one embodiment, synchronizing the stock data and all the first incremental data in the source system to the target system is achieved through the following steps, including:

[0043] Step a1: If the target system is verified to meet the read and write conditions, synchronize part of the stock data in the source system to the target system.

[0044] Step a2: If the synchronization of the part of the stock data is successful, determine that the target system meets the data migration conditions, and sequentially synchronize the remaining stock data and all the first incremental data in the source system to the target system.

[0045] This embodiment aims to determine the accuracy of data migration by viewing the synchronization result of this part of the stock data. For example, determine the accuracy of data transmission, the accuracy of data written to the target system after being transmitted to the target system, etc. If the synchronization of this part of the stock data is successful, it means that the data migration between the source system and the target system has high accuracy, that is, the target system meets the data migration conditions. At this time, sequentially synchronizing the remaining stock data and all the first incremental data in the source system to the target system can ensure the accuracy of the migration of the remaining stock data and the first incremental data.

[0046] It can be understood that if the synchronization of this part of the stock data fails, stop the data migration, roll back to the single-write state of the source system, and find the reason for the data synchronization failure.

[0047] In the technical solution of the data migration method provided by the embodiment of the present invention, from the moment when the first instruction is generated to the start of data migration, the first incremental data is written to the source system and the target system in a dual-write mode; if the target system is verified to meet the read-write conditions, it is defaulted that the read-write performance of the target system is good during data migration, that is, the second incremental data generated by the data generation device during data migration can be accurately written to the target system, and the second incremental data written to the target system is readable. Therefore, only the stock data and the first incremental data generated before data migration need to be synchronized to the target system; compared with the synchronization difficulty of real-time data, the synchronization difficulty of non-real-time data is lower and the synchronization speed is also faster. Therefore, the first incremental data and the stock data generated before data migration can be quickly synchronized from the source system to the target system; compared with first synchronizing the data to be migrated from the source system to the target file and then synchronizing the data to be migrated from the target file to the target system, directly synchronizing the stock data and the first incremental data from the source system to the target system further improves the data migration speed.

[0048] Figure 2 It is another flowchart of the data migration method provided by the embodiment of the present invention. This embodiment is used to add steps for re-migrating the stock data and / or the first incremental data that failed to migrate on the basis of the foregoing embodiment, as Figure 2 shown, the method includes:

[0049] S210. In response to the first instruction, write the first incremental data generated by the data generation device to the source system and the target system.

[0050] S220. If the target system is verified to meet the read-write conditions, synchronize the stock data in the source system and all the first incremental data to the target system, where the stock data is the target data written to the source system before the generation moment of the first instruction, and the first incremental data is the target data written to the source system between the generation moment of the first instruction and the start moment of synchronizing the stock data.

[0051] S230. If there is stock data and / or first incremental data for which the synchronization fails, write the stock data and / or first incremental data for which the synchronization fails to the target message queue.

[0052] In one embodiment, the stock data and / or first incremental data for which the synchronization fails are determined through the following steps, including:

[0053] Step b1. For each target data in the stock data and all the first incremental data, determine the consistency between the current target data and the data of the current target data synchronized to the target system.

[0054] Optionally, calculate the MD5 values of the current target data and the data that the current target data has been synchronized to the target system respectively. If the MD5 values of the two are the same, it is considered that the two are the same and consistent, that is, the synchronization of the current target data is successful.

[0055] Step b2: If they are inconsistent, generate synchronization exception information for the current target data.

[0056] If the MD5 values of the two are different, it indicates that the synchronization of the current target data fails, and synchronization exception information for the current target data is generated, so that the user can view the reason for the data synchronization failure according to the synchronization exception information, and manually complete the synchronization of the current target data based on the synchronization exception information.

[0057] In one embodiment, if the generation speed of the synchronization exception information meets the set speed condition, an alarm message is output, the data migration is interrupted, and the source system single-write operation is triggered, so that the target data generated by the data generation device is only written into the source system. This embodiment realizes determining the number of target data with synchronization failures per unit time through the generation speed of the synchronization exception information, thereby determining the stability of data synchronization. In the case of low data synchronization stability, an alarm message is output, the data migration is interrupted, and the source system single-write operation is triggered. The alarm message and the interrupted data migration allow the user to view the cause of the failure in a timely manner and prevent more target data with synchronization failures from occurring; since the second incremental data is not synchronized, it is necessary to ensure the stability and accuracy of the write of the second incremental data. Triggering the source system single-write operation can prevent the occurrence of abnormal write of the second incremental data due to a failure of the target system. After determining that the target system is normal, the dual-write can be restarted and the migration of the remaining data can be continued.

[0058] After the first stock data and / or incremental data with synchronization failures are determined, they are added to the target message queue. Managing the stock data and / or the first incremental data with synchronization failures through the target message queue improves the orderliness of the management of the stock data and / or the first incremental data with synchronization failures.

[0059] S240: Synchronize the stock data and / or the first incremental data in the target message queue to the target system.

[0060] By regularly synchronizing the stock data and / or the first incremental data in the target message queue to the target system, the re-migration of the stock data and / or the first incremental data with synchronization failures is automatically completed. Compared with manually processing the stock data and / or the first stock data with synchronization failures, completing the re-migration of the stock data and / or the first incremental data with synchronization failures through the target message queue can improve the accuracy and speed of the re-migration of the stock data and / or the first incremental data with synchronization failures.

[0061] In one embodiment, after the first re - migration of the stock data and / or the first incremental data with synchronization failures, if there are still stock data and / or first incremental data with synchronization failures, the stock data and / or first incremental data with re - synchronization failures are taken as the stock data and / or first incremental data with synchronization failures, and the step of writing the stock data and / or incremental data with synchronization failures into the target message queue is returned until the stock data and / or the first incremental data with synchronization failures are synchronized to the target system, or the synchronization times reach the set number of times condition, and then the synchronization of the stock data and / or stock data with synchronization failures is stopped, and corresponding exception information is generated.

[0062] In the embodiment of the present invention, by first migrating part of the stock data to verify whether the target system meets the data migration conditions, and in the case that it meets the data migration conditions, the remaining stock data and all the first incremental data are synchronized to the target system. By the way of first verification and then transmission, the accuracy of synchronizing the remaining stock data and all the first incremental data is determined.

[0063] Figure 3 It is another flowchart of the data migration method provided by the embodiment of the present invention. This embodiment is used to refine the specific synchronization method of the stock data and the first incremental data in the foregoing embodiment, such as Figure 3 As shown, the method includes:

[0064] S310. In response to the first instruction, write the first incremental data generated by the data generation device into the source system and the target system.

[0065] S320. If the target system is verified to meet the read - write conditions, then in a distributed concurrent data migration manner, synchronize the stock data and all the first incremental data in the source system to the target system. The stock data is the target data written into the source system before the generation time of the first instruction, and the first incremental data is the target data written into the source system between the generation time of the first instruction and the start time of synchronizing the stock data.

[0066] When synchronizing the stock data and all the first incremental data in the source system to the target system in a distributed concurrent data migration manner, the number of distributed data migration tasks can be set by itself according to the performance of the server device and the data migration speed requirements, such as 3, 4, or 5, etc.

[0067] In one embodiment, for the stock data, each distributed data migration task locks the target data in the current batch read in a preemptive manner and synchronizes the locked target data to the target system; for the first incremental data, each distributed data migration task locks the target data in the current batch read in a preemptive manner and synchronizes the locked target data to the target system. Exemplarily, the number of distributed data migration tasks is 5; for the stock data, the stock data is migrated in batches. Taking the example that each batch of stock data includes 2,000 target data. For each of the 5 distributed data migration tasks, when it is queried that the status of any target data is not migrated, the target data is locked in a preemptive manner and the target data is synchronized to the target system. Each distributed data migration task prevents other distributed data migration tasks from performing data synchronization operations on it by locking the target data, avoiding the situation of duplicate migration of the target data and improving the efficiency of data migration. Regarding the method of locking the target data, lock forms such as optimistic locks can be used to lock the target data.

[0068] In one embodiment, if the target system is verified to meet the read and write conditions, a timed data migration task is generated; when the timed moment corresponding to the timed data migration task arrives, a data migration instruction is generated; in response to the data migration instruction, the stock data and all the first incremental data in the source system are synchronized to the target system in a distributed concurrent data migration manner. In this embodiment, timed data migration is realized through the timed data migration task, improving the flexibility of the data migration time selection.

[0069] The embodiment of the present invention improves the data migration speed through the distributed concurrent data migration method.

[0070] Figure 4 It is another flowchart of the data migration method provided by the embodiment of the present invention. This embodiment is used to add a double query step on the basis of the foregoing embodiment, as Figure 4 shown, the method includes:

[0071] S410. In response to the first instruction, write the first incremental data generated by the data generation device into the source system and the target system.

[0072] S4101. In response to the data query instruction, query the target data corresponding to the data query instruction in the target system.

[0073] In the case of detecting a data query instruction sent by the front-end application, first query the target data corresponding to the data query instruction in the target system. If the target data corresponding to the data query instruction exists in the target system, output the target data; if the target data corresponding to the data query instruction does not exist in the target system, execute S4102.

[0074] S4102: If the query fails, query the target data corresponding to the data query instruction in the source system, output the query result, and generate a target system query failure message.

[0075] Since the source system runs stably and includes the target data of all users, usually the target data corresponding to the data query instruction can be queried in the source system. After locating the target data in the source system, send the target data to the front-end application. If the target data corresponding to the data query instruction is not queried in the source system, generate a corresponding prompt message. Whether or not the target data corresponding to the data query instruction is queried in the source system, a target system query failure message is generated.

[0076] The target system query failure message is used to record the reason why the target data corresponding to the data query instruction cannot be read in the target system. For example, the target system does not store the target data corresponding to the data query instruction, or the target system stores the target data corresponding to the data query instruction but cannot read the target data, that is, there is a data reading problem in the target system.

[0077] After the target system query failure message is generated, the user can determine the reason for the data query failure based on the target system query failure message and solve the corresponding problem in time.

[0078] It can be understood that regardless of whether the target data corresponding to the data query instruction can be queried in the target system, since all users' target data is stored in the source system, finally the target data corresponding to the data query instruction can be provided. Therefore, through the double-write operation, double-read mode, and target system query failure message, it can be completed whether the target system meets the read and write conditions without the user's awareness. When the target system meets the read and write conditions, execute S420. When the target system does not meet the read and write conditions, solve the corresponding read and write problems in time to improve the read and write accuracy and stability of the target system.

[0079] S420: If the target system is verified to meet the read and write conditions, synchronize the stock data and all first incremental data in the source system to the target system. The stock data is the target data written into the source system before the generation time of the first instruction, and the first incremental data is the target data written into the source system between the generation time of the first instruction and the start time of the stock data synchronization.

[0080] In the embodiment of the present invention, the read-write verification of the target system is completed without the user's awareness through the dual-read mode and the dual-write operation, providing a high user experience.

[0081] Figure 5 It is another flowchart of the data migration method provided by the embodiment of the present invention. In this embodiment, the source system is a storage system based on Hbace, the target system is an Object Storage Service (OSS) system, and the migrated data (target data) is a settlement certificate file. The technical solution will be described in detail below by taking the settlement certificate file as an example. The settlement certificate file can be used to prove that the borrower or lender has repaid the loan, and there is no debt relationship between the borrower or lender and the lender. The settlement certificate file is usually applied for by the user to the platform. After the platform verifies that the loan has been repaid, the settlement certificate file of the applicant is generated by the data generation device. It can be understood that for users with multiple loans, they may apply for multiple settlement certificate files at different time points. For example, user A applied for the first loan the year before last and repaid the first loan the year before last, and applied for a settlement certificate file after the first loan was repaid; user A applied for the second loan last year and repaid the second loan last year, and applied for a settlement certificate file after the second loan was repaid. In this way, there are two settlement certificate files from different periods in the user's account, and both of these two settlement certificate files are migrated data.

[0082] As Figure 5 shown, the method includes:

[0083] S510. In response to the first instruction, write the first incremental data generated by the data generation device into the source system and the target system.

[0084] The user sends a request for applying for a settlement certificate file to the platform through the front-end application. The platform performs verification of set parameters according to this request. For example, user identity verification, the number of settlement certificate files applied for in the current month, whether the service corresponding to the settlement certificate file is opened, verification of outstanding repayment information, credit check, etc. After the verification of the set parameters passes, the corresponding parameters are written into the application flow table, and a corresponding message body is generated; the message body is added to the first message queue. The message body includes information such as an application serial number, a user identifier, and contact information. The contact information can be an email address and / or a telephone number, etc.

[0085] The platform monitors the first message queue in real time through a data generation device. After a message body is monitored, the message body is converted into a corresponding proof application data entity, and the integrity of the necessary information of the proof application data entity is verified. If all the necessary information in the proof application data entity is complete, a settlement proof template file is obtained, and each data in the proof application data entity is added to the corresponding position of the settlement proof template file to generate a file to be signed; a signature service is called to generate a settlement proof file in PDF format.

[0086] The dual-write mode is enabled, and the generated settlement proof file is written to both the source system and the target system by the platform. If the write is abnormal, an exception message is directly thrown, message automatic retry alarm is performed, manual access is performed for troubleshooting, and the operation is rolled back to single write in the source system, and the target system is closed; if the write is normal, it is observed for a period of time, such as observing for 3 days, to check whether the write performance of the target cluster is stable and whether tp999 (the minimum time required to satisfy 99.9‰ of network requests) fluctuates. If both the write performance and tp999 are stable, it proves that the target system is stable in terms of data writing.

[0087] Among them, write anomalies include inability to write or write errors. In one embodiment, after the settlement proof file is written to the source system and the target system, it is determined whether the write results of the source system and the target system are consistent. If they are inconsistent, it is considered that the write to the target system is abnormal.

[0088] S520. If the target system is verified to meet the read and write conditions, the stock data and all first incremental data in the source system are synchronized to the target system by means of distributed concurrent data migration.

[0089] On the one hand, since the realization of the platform function depends on the cooperation of multiple server devices, during the process of switching to dual-write online, it is possible that the settlement proof files generated by some online server devices are written to both the source system and the target system at the same time, and the settlement proof files generated by offline server devices are only written to the source system. For a data query request sent by a front-end application, if only the target system is queried, the corresponding settlement proof file may not be queried. On the other hand, there may be a read failure in the target system, that is, the settlement proof file can be correctly written to the target system, but cannot be read from the target system. Therefore, after the dual-write operation is switched, the dual-read mode is enabled.

[0090] After the dual-read mode is enabled, first query the settlement certificate file corresponding to the data query request in the target system. If it can be queried, read the settlement certificate file and send it to the front-end application. If it cannot be queried, query the settlement certificate file corresponding to the data query instruction in the source system. If the corresponding settlement certificate file is queried, send the settlement certificate file to the front-end application and generate a target system query failure message at the same time. If the settlement certificate file corresponding to the data query instruction cannot be queried in the source system either, return a prompt message indicating that the data does not exist and generate a target system query failure message at the same time.

[0091] The target system query failure message is convenient for users to clarify the reason for the failure to query the settlement certificate file corresponding to the data query instruction in the target system. In this way, users can take corresponding measures according to the specific reasons to improve the read and write stability and accuracy of the target system.

[0092] If the target system is stable in writing and reading within the set time period, it is determined that the target system is verified to meet the read and write conditions. Exemplarily, the set time period is 3 days. No write exception prompt message within this set time period indicates that the target system is stable in writing; at least one settlement certificate file corresponding to the target system query failure message does not belong to the first incremental data, indicating that the target system is stable in reading.

[0093] When the target system is verified to meet the read and write conditions, start the migration operation of the stock data and the first incremental data. First, obtain the source bucket address in the source system for storing the settlement certificate file and the target bucket address in the target system for storing the settlement certificate file. Each distributed data migration task reads the corresponding settlement certificate file from the source bucket address of the source system and then writes the read settlement certificate file to the target bucket address of the target system. Among them, a bucket can be understood as a storage container. All objects belong to a certain storage space, and the access rights, life cycle, etc. can be controlled by modifying and setting the storage space attributes, which directly act on all objects within this storage space.

[0094] Set the start and end times of the data migration, the number of distributed tasks, and the number of settlement certificate files included in each batch of migrated data through the timing scheduling system. Specifically, it can be set through the cron (scheduled task) expression.

[0095] In one embodiment, the time when the source system writes the first data is set as January 1st, year M, the time point when the dual-write mode is enabled is May 5th, year M + 1, and the time point when data migration is triggered is May 12th, year M + 1. Therefore, the creation time of the stock data is from January 1st, year M to May 5th, year M + 1, the creation time of the first incremental data is from May 5th, year M + 1 to May 12th, year M + 1, and the time range of the migrated data is on May 12th, year M + 1. Since the first incremental data generated during the period from May 5th, year M + 1 to May 12th, year M + 1 is written to both the source system and the target system simultaneously, although there may be problems with unstable write performance in the early stage of this period in the target system, resulting in some of the first incremental data being written fails, most of the first incremental data generated by the platform during this period will still be written to the target system. In this embodiment, when writing the first incremental data read from the source system to the target system, if it is found that the target system already stores the same first incremental data, it is directly overwritten. By migrating the first incremental data, it can be ensured that the target system stores all the target data in the source system.

[0096] S530. For each target data in the stock data and all the first incremental data, determine the consistency between the current target data and the data of the current target data synchronized to the target system.

[0097] Take the settlement certificate file to be written to the target system as the target settlement certificate file, calculate the MD5 values of this settlement certificate file and the target settlement certificate file respectively, and then determine whether these two values are consistent. If they are consistent, it indicates that the migration of this settlement certificate file is successful; if they are inconsistent, it indicates that the migration of this settlement certificate file fails.

[0098] S540. If they are inconsistent, generate synchronization exception information for the current target data, or add the current target data to the target message queue.

[0099] In one embodiment, if the migration of any settlement certificate file fails, generate synchronization exception information for this settlement certificate file, and the user manually migrates this settlement certificate file according to this synchronization exception information.

[0100] In one embodiment, if the migration of any settlement certificate file fails, add this settlement certificate file to the target message queue and execute S570.

[0101] S550. Synchronize the target data in the target message queue to the target system.

[0102] In one embodiment, the target message queue is monitored in real time. When a settlement certificate file in the target message queue is monitored, the settlement certificate file is synchronized to the target system, so as to achieve the technical effect of automatically re-migrating the settlement certificate file with a synchronization failure.

[0103] In one embodiment, the target message queue is scanned regularly. When it is determined that the target message queue includes at least one settlement certificate file, the at least one settlement certificate file is synchronized to the target system, so as to achieve batch re-migration of the settlement certificate file with a synchronization failure and improve the speed of re-migrating the settlement certificate file with a synchronization failure.

[0104] In one embodiment, if any settlement certificate file in the target message queue fails to be re-migrated, it is re-added to the target message queue, then the settlement certificate file is read from the target message queue and written to the target system again. This is repeated until the settlement certificate file is written to the target system or the number of repeated writes reaches the set number of times. For the latter case, the operation of re-migrating it is stopped, and corresponding synchronization exception information is generated, so that the user can manually migrate the settlement certificate file with a synchronization failure according to the synchronization exception information. This embodiment realizes the technical effect of automatically and accurately completing data migration through a combination of automatic re-migration and manual migration, and improves the accuracy and flexibility of data migration through manual intervention when necessary.

[0105] In one embodiment, after the migration of the stock data and the incremental data is completed, the write permission and the query permission of the source system are closed, so that the settlement certificate files generated by the platform are only written to the target system, and the platform only queries the corresponding settlement certificate files from the target system.

[0106] The embodiments of the present invention improve the migration speed of the stock data and the first incremental data through a distributed concurrent data migration method; by adding the stock data and / or the first incremental data with a migration failure to the target message queue, and re-migrating the stock data and / or the first incremental data with a synchronization failure in the target message queue, the technical effect of re-migrating the target data with a migration failure is achieved; by verifying the consistency of each target data before and after migration, and generating corresponding synchronization exception information when they are inconsistent, the real-time monitoring of the migration process is achieved; by directly synchronizing the stock data and the first incremental data generated before data migration from the source system to the target system, a high data migration speed and accuracy can be ensured.

[0107] The following are embodiments of a data migration device provided by the embodiments of the present invention. This device and the data migration method in the above Embodiment 1 belong to the same inventive concept. For the details not described in detail in the embodiments of the data migration device, reference may be made to the content of the above embodiments.

[0108] Figure 6A The structural schematic diagram of the data migration device provided by the embodiment of the present invention. As Figure 6A shown, the device includes:

[0109] A dual-write module 610, configured to write the first incremental data generated by the data generation device into the source system and the target system in response to a first instruction;

[0110] A data migration module 620, configured to synchronize the stock data and all the first incremental data in the source system to the target system if the target system is verified to meet the read / write conditions, where the stock data is the target data written into the source system before the generation time of the first instruction, and the first incremental data is the target data written into the source system between the generation time of the first instruction and the start time of the stock data synchronization.

[0111] In one embodiment, the data migration module 620 is specifically configured to:

[0112] If the target system is verified to meet the read / write conditions, synchronize part of the stock data in the source system to the target system;

[0113] If the synchronization of the part of the stock data is successful, determine that the target system meets the data migration conditions, and synchronize the remaining stock data and all the first incremental data in the source system to the target system in sequence.

[0114] In one embodiment, as Figure 6B shown, the device further includes a re-migration module 630, and the re-migration module 630 is configured to:

[0115] If there is stock data and / or first incremental data with synchronization failure, write the stock data and / or first incremental data with synchronization failure into the target message queue;

[0116] Synchronize the stock data and / or first incremental data in the target message queue to the target system.

[0117] In one embodiment, the data migration module 620 is specifically configured to:

[0118] Adopt a distributed concurrent data migration method to synchronize the stock data and all the first incremental data in the source system to the target system.

[0119] In one embodiment, the data migration module 620 is specifically configured to:

[0120] For the stock data, each distributed data migration task locks and reads the target data in the current batch in a preemptive manner, and synchronizes the locked target data to the target system;

[0121] For the first incremental data, each distributed data migration task locks the target data in the current batch read in a preemptive manner and synchronizes the locked target data to the target system.

[0122] In one embodiment, the re-migration module 630 determines the stock data and / or the first incremental data for which synchronization fails through a verification unit, where the verification unit is configured to:

[0123] For each target data in the stock data and all the first incremental data, determine the consistency between the current target data and the data in which the current target data is synchronized to the target system;

[0124] If they are inconsistent, generate synchronization exception information for the current target data.

[0125] In one embodiment, the re-migration module 630 is further configured to:

[0126] If the generation speed of the synchronization exception information meets the set speed condition, output an alarm message, interrupt the data migration, and trigger a single write operation of the source system, so that the target data generated by the data generation device is only written into the source system.

[0127] In one embodiment, as Figure 6C shown, the device further includes a dual query module 640, and the dual query module is configured to:

[0128] In response to a data query instruction, query the target data corresponding to the data query instruction in the target system;

[0129] If the query fails, query the target data corresponding to the data query instruction in the source system, output the query result, and generate a target system query failure information.

[0130] In one embodiment, the target data is a settlement certificate file.

[0131] In the technical solution of the data migration device provided by the embodiment of the present invention, from the moment when the first instruction is generated to the start of data migration, the first incremental data is written to the source system and the target system in a dual-write mode; if the target system is verified to meet the read-write conditions, it is defaulted that the read-write performance of the target system is good during data migration, that is, the second incremental data generated by the data generation device during data migration can be accurately written to the target system, and the second incremental data written to the target system is readable. Therefore, only the stock data and the first incremental data generated before data migration need to be synchronized to the target system; compared with the synchronization difficulty of real-time data, the synchronization difficulty of non-real-time data is lower and the synchronization speed is also faster. Therefore, the first incremental data and the stock data generated before data migration can be quickly synchronized from the source system to the target system; compared with first synchronizing the data to be migrated from the source system to the target file and then synchronizing the data to be migrated from the target file to the target system, directly synchronizing the stock data and the first incremental data from the source system to the target system further improves the data migration speed.

[0132] The data migration device provided by the embodiment of the present invention can execute the data migration method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the data migration method.

[0133] Figure 7 It is a schematic structural diagram of a server device provided by an embodiment of the present invention. Figure 7 It shows a block diagram of an exemplary server device 12 suitable for implementing the embodiments of the present invention. Figure 7 The shown server device 12 is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0134] As Figure 7 shown, the server device 12 is presented in the form of a general-purpose computing device. The components of the server device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0135] The bus 18 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0136] Server device 12 typically includes a variety of computer system readable media. These media can be any available media accessible to server device 12, including volatile and non-volatile media, removable and non-removable media.

[0137] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Server device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing on non-removable, non-volatile magnetic media (not shown in FIG. 6, commonly referred to as a "hard disk drive"). Although not shown in FIG. 6, a disk drive can be provided for reading and writing on removable non-volatile disks (such as "floppy disks"), and an optical disk drive for reading and writing on removable non-volatile optical disks (such as CD-ROM, DVD-ROM or other optical media). In these cases, each drive can be connected to bus 18 through one or more data media interfaces. System memory 28 can include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0138] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods in the embodiments described in the present invention.

[0139] Server device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the server device 12, and / or communicate with any device that enables the server device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 22. Also, server device 12 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. As shown, network adapter 20 communicates with other modules of server device 12 through bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with server device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0140] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the steps of the data migration method provided in the embodiments of the present invention. The method includes:

[0141] In response to a first instruction, write the first incremental data generated by the data generation device to the source system and the target system;

[0142] If the target system is verified to meet the read / write conditions, synchronize the stock data in the source system and all the first incremental data to the target system. The stock data is the target data written to the source system before the generation time of the first instruction, and the first incremental data is the target data written to the source system between the generation time of the first instruction and the start time of the synchronization of the stock data.

[0143] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the data migration method provided in any embodiment of the present invention.

[0144] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps of the data migration method provided in the foregoing embodiments of the present invention. The method includes:

[0145] In response to a first instruction, write the first incremental data generated by the data generation device to the source system and the target system;

[0146] If the target system is verified to meet the read / write conditions, synchronize the stock data in the source system and all the first incremental data to the target system. The stock data is the target data written to the source system before the generation time of the first instruction, and the first incremental data is the target data written to the source system between the generation time of the first instruction and the start time of the synchronization of the stock data.

[0147] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable media may be computer-readable signal media or computer-readable storage media. The computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, the computer-readable storage media may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component.

[0148] The computer-readable signal media may include data signals propagated in a baseband or as part of a carrier wave, which carry computer-readable program codes. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal media may also be any computer-readable medium other than the computer-readable storage media, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or component.

[0149] The program codes contained on the computer-readable media can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0150] The computer program codes for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program codes can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0151] Those of ordinary skill in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0152] Note that the above is only the preferred embodiment of the present invention and the applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data migration method, characterized in that, the method includes: In response to a first instruction, write first incremental data generated by a data generation device to a source system and a target system; If the target system is verified to meet the read / write conditions, synchronize the stock data in the source system and all the first incremental data to the target system; Wherein, the stock data is the target data written to the source system before the generation time of the first instruction, and the first incremental data is the target data written to the source system between the generation time of the first instruction and the start time of stock data synchronization.

2. The method according to claim 1, characterized in that, the step of if the target system is verified to meet the read / write conditions, synchronize the stock data in the source system and all the first incremental data to the target system includes: If the target system is verified to meet the read / write conditions, synchronize part of the stock data in the source system to the target system; If the synchronization of the part of the stock data is successful, determine that the target system meets the data migration conditions, and sequentially synchronize the remaining stock data and all the first incremental data in the source system to the target system.

3. The method according to claim 1, characterized in that, further includes: If there is stock data and / or first incremental data for which synchronization fails, write the stock data and / or first incremental data for which synchronization fails to a target message queue; Synchronize the stock data and / or first incremental data in the target message queue to the target system.

4. The method according to claim 1, characterized in that, the step of synchronize the stock data in the source system and all the first incremental data to the target system includes: Adopt a distributed concurrent data migration method to synchronize the stock data in the source system and all the first incremental data to the target system.

5. The method according to claim 4, characterized in that, the step of adopt a distributed concurrent data migration method to synchronize the stock data in the source system and all the first incremental data to the target system includes: For the stock data, each distributed data migration task locks and reads the target data in the current batch in a preemptive manner, and synchronizes the locked target data to the target system; For the first incremental data, each distributed data migration task locks and reads the target data in the current batch in a preemptive manner, and synchronizes the locked target data to the target system.

6. The method according to claim 3, characterized in that, Determine the stock data and / or first incremental data for which synchronization fails through the following steps: For each target data in the stock data and all the first incremental data, determine the consistency between the current target data and the data to which the current target data is synchronized to the target system; If they are inconsistent, generate synchronization exception information for the current target data.

7. The method according to claim 6, characterized in that, further includes: If the generation speed of the synchronization exception information meets the set speed condition, an alarm message is output, data migration is interrupted, and a single-write operation of the source system is triggered, so that the target data generated by the data generation device is only written into the source system.

8. The method according to claim 1, wherein, after writing the first incremental data generated by the data generation device into the source system and the target system in response to the first instruction, further comprising: in response to a data query instruction, querying for target data corresponding to the data query instruction in the target system; if the query fails, querying for the target data corresponding to the data query instruction in the source system, outputting a query result, and generating a target system query failure message.

9. The method according to claim 1, wherein, the target data is a settlement certificate document.

10. A data migration device, wherein, comprising: a dual-write module for writing the first incremental data generated by the data generation device into the source system and the target system in response to the first instruction; a data migration module for synchronizing the stock data in the source system and all the first incremental data to the target system if the target system is verified to meet the read / write conditions, where the stock data is the target data written into the source system before the generation time of the first instruction, and the first incremental data is the target data written into the source system between the generation time of the first instruction and the start time of the stock data synchronization.

11. A server device, wherein, the server device comprises: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the data migration method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, wherein, when the program is executed by a processor, it implements the data migration method according to any one of claims 1-9.

Images