Data migration method, device, terminal equipment and storage medium

Abstract

The present application discloses a data migration method, apparatus, terminal device and storage medium, wherein the data migration method includes: continuously monitoring the data table of the source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table; if the data table of the source system is monitored to change, the corresponding data table change information is obtained; according to the data table change information, the target data field corresponding to the changed data table is determined from a plurality of data fields; and the data in the target data field is migrated from the source system to the target system. Based on the scheme of the present application, a migration method based on data fields is adopted, and by continuously monitoring the changes in the data table of the source system, migration is performed only when the data table changes. In this way, incremental migration can be achieved, which reduces the impact on the business while reducing performance overhead, and improves the efficiency and reliability of data migration.

Description

Technical Field

[0001] The present application relates to the technical field of data migration, and in particular to a data migration method, apparatus, terminal device and storage medium. Background Art

[0002] For data integration, upgrade, backup and other needs, it is usually necessary to migrate data from the source system to the target system. The traditional data migration method is to migrate a large amount of existing data from the source system to the target system in a table-by-table manner within a specific time window, that is, all data is re-migrated each time, including the data that has been migrated, which leads to unnecessary performance overhead. In addition, the traditional data migration method requires a longer time window to complete the migration. Providing the time window required for data migration by downtime may cause business interruption and user dissatisfaction. In addition, the existing technical solutions often ignore the situation where the data objects to be migrated are physically deleted or the target data warehouse of the migrated data is physically deleted, resulting in task execution failure and even system data confusion.

[0003] In summary, current data migration methods have high performance overhead and affect business. Summary of the invention

[0004] The main purpose of this application is to provide a data migration method, apparatus, terminal device and storage medium, aiming to improve the efficiency and accuracy of the data migration method and enhance the availability of the data migration system.

[0005] To achieve the above object, the present application provides a data migration method, which includes:

[0006] Continuously monitoring a data table of a source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table;

[0007] If a change is detected in the data table of the source system, the corresponding data table change information is obtained;

[0008] According to the data table change information, determining a target data field corresponding to the changed data table from the plurality of data fields;

[0009] Migrate the data in the target data domain from the source system to the target system.

[0010] Optionally, before the step of migrating the data in the target data domain from the source system to the target system, the step further includes:

[0011] Acquire source domain root information corresponding to the target data domain, and target domain root information corresponding to the source domain root information in the target system;

[0012] According to the source domain root information, determining whether there is a source domain root in the target data domain;

[0013] If the source domain root does not exist in the target data domain, the target domain root in the target system is deleted according to the target domain root information.

[0014] Optionally, after the step of migrating the data in the target data domain from the source system to the target system, the step further includes:

[0015] Based on the preset heartbeat monitoring events at a preset time frequency, the data table change information and the key detailed indicators corresponding to the target data field are obtained;

[0016] Generate a number of visualization components based on the data table change information, and display them on a pre-set data cabin front end;

[0017] According to the key detail indicators, obtaining the first type of key detail data in the target data field from the source system, and obtaining the second type of key detail data in the target data field from the target system;

[0018] The first-category key detailed data and the second-category key detailed data are compared to obtain comparison result information.

[0019] Optionally, the method further comprises:

[0020] Acquire data change track information by pre-generating a data plug-in, wherein the data change track information includes data domain change information, data field change information and the data table change information;

[0021] Performing data standardization processing on the data domain change information, the data field change information and the data table change information to obtain standardized change trajectory information;

[0022] The standardized change trajectory information is asynchronously stored in a preset trajectory information database.

[0023] Optionally, the source system is provided with a source database, the source database stores data corresponding to the plurality of data fields, and the step of continuously monitoring the data table of the source system includes:

[0024] Continuously acquiring the transaction log files of the source database, and parsing the transaction log files to obtain data table operation information;

[0025] The data table of the source system is continuously monitored according to the data table operation information.

[0026] Optionally, after the step of acquiring corresponding data table change information if a change in the data table of the source system is monitored, and before the step of determining, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields, further comprising:

[0027] Generate a message corresponding to the data table change information, and store the message corresponding to the data table change information in a preset distributed publish-subscribe message kafka system;

[0028] If a message corresponding to the data table change information published by the kafka system is monitored, the message corresponding to the data table change information is parsed to obtain the restored data table change information;

[0029] The step of determining, from the plurality of data fields, a target data field corresponding to the changed data table according to the data table change information comprises:

[0030] According to the restored data table change information, a target data field corresponding to the changed data table is determined from the plurality of data fields.

[0031] Optionally, the number of the target data fields is several, and the step of migrating the data in the target data fields from the source system to the target system includes:

[0032] Analyze the hierarchical relationship of the target data field to determine the root node corresponding to the target data field;

[0033] Generate corresponding asynchronous tasks according to the root nodes corresponding to the target data fields respectively, and store the asynchronous tasks in a preset asynchronous task queue;

[0034] Based on the plurality of asynchronous tasks, data corresponding to each of the plurality of target data fields is obtained through a preset Java persistence interface, and the data corresponding to the target data fields is migrated from the source system to the target system.

[0035] The present application also provides a data migration device, which includes:

[0036] A monitoring module, used for continuously monitoring a data table of a source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table;

[0037] An acquisition module, configured to acquire corresponding data table change information if a change in the data table of the source system is detected;

[0038] A determination module, configured to determine, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields;

[0039] A migration module is used to migrate the data in the target data domain from the source system to the target system.

[0040] An embodiment of the present application further proposes a terminal device, which includes a memory, a processor, and a data migration program stored in the memory and executable on the processor, wherein the data migration program implements the steps of the data migration method described above when executed by the processor.

[0041] An embodiment of the present application further provides a computer-readable storage medium, on which a data migration program is stored. When the data migration program is executed by a processor, the steps of the data migration method described above are implemented.

[0042] The data migration method, apparatus, terminal device and storage medium proposed in the embodiments of the present application continuously monitor the data table of the source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table; if a change in the data table of the source system is monitored, the corresponding data table change information is obtained; based on the data table change information, the target data field corresponding to the changed data table is determined from the plurality of data fields; and the data in the target data field is migrated from the source system to the target system. Based on the scheme of the present application, a migration method based on data fields is adopted, by continuously monitoring the changes in the data table of the source system, and migration is performed only when the data table changes. In this way, incremental migration can be achieved, which reduces the impact on the business while reducing performance overhead, and improves the efficiency and reliability of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the data migration device of this application belongs;

[0044] Figure 2 This is a flowchart of the first exemplary embodiment of the data migration method of this application;

[0045] Figure 3 A schematic diagram of a physical deletion verification and processing flow provided for the data migration method of this application;

[0046] Figure 4 A schematic diagram of a data statistics flow during a data migration process provided by the data migration method of this application;

[0047] Figure 5 A schematic diagram of a data monitoring process during data migration provided by the data migration method of the present application;

[0048] Figure 6 This is a flow chart of a second exemplary embodiment of the data migration method of this application;

[0049] Figure 7 This is a flowchart of a third exemplary embodiment of the data migration method of this application;

[0050] Figure 8 This is a flowchart of a fourth exemplary embodiment of the data migration method of this application;

[0051] Fig. 9 This is a flowchart of a fifth exemplary embodiment of the data migration method of this application;

[0052] Fig.10 This is a flowchart of a sixth exemplary embodiment of the data migration method of this application;

[0053] Fig.11 This is a schematic diagram of the module results of the data migration device according to an embodiment of the present application.

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The main solution of the embodiment of the present application is: continuously monitor the data table of the source system, wherein the source system includes several preset data fields, and any data field includes at least one corresponding data table; if the data table of the source system is monitored to change, the corresponding data table change information is obtained; according to the data table change information, the target data field corresponding to the changed data table is determined from the several data fields; and the data of the target data field is migrated from the source system to the target system. Based on the scheme of the present application, a migration method based on data fields is adopted, by continuously monitoring the changes of the data table of the source system, and migration is only performed when the data table changes. In this way, incremental migration can be achieved, which reduces the impact on the business while reducing performance overhead, and improves the efficiency and reliability of data migration.

[0057] Specifically, refer to Figure 1 , Figure 1This is a functional module diagram of the terminal device to which the data migration device of this application belongs. The data migration device can be a device that is independent of the terminal device and can perform data migration, and it can be carried on the terminal device in the form of hardware or software. The terminal device can be a smart mobile terminal with data processing functions such as a mobile phone or a tablet computer, or a fixed terminal device or server with data processing functions.

[0058] In this embodiment, the terminal device to which the data migration apparatus belongs at least includes an output module 110 , a processor 120 , a memory 130 and a communication module 140 .

[0059] The memory 130 stores an operating system and a data migration program, and the data migration device can store monitoring information, data table change information, and other information in the memory 130; the output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with an external device or server through the communication module 140.

[0060] When the data migration program in the memory 130 is executed by the processor, the following steps are implemented:

[0061] Continuously monitoring a data table of a source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table;

[0062] If a change is detected in the data table of the source system, the corresponding data table change information is obtained;

[0063] According to the data table change information, determining a target data field corresponding to the changed data table from the plurality of data fields;

[0064] Migrate the data in the target data domain from the source system to the target system.

[0065] Furthermore, when the data migration program in the memory 130 is executed by the processor, the following steps are also implemented:

[0066] Acquire source domain root information corresponding to the target data domain, and target domain root information corresponding to the source domain root information in the target system;

[0067] According to the source domain root information, determining whether there is a source domain root in the target data domain;

[0068] If the source domain root does not exist in the target data domain, the target domain root in the target system is deleted according to the target domain root information.

[0069] Furthermore, when the data migration program in the memory 130 is executed by the processor, the following steps are also implemented:

[0070] Based on the preset heartbeat monitoring events at a preset time frequency, the data table change information and the key detailed indicators corresponding to the target data field are obtained;

[0071] Generate a number of visualization components based on the data table change information, and display them on a pre-set data cabin front end;

[0072] According to the key detail indicators, obtaining the first type of key detail data in the target data field from the source system, and obtaining the second type of key detail data in the target data field from the target system;

[0073] The first-category key detailed data and the second-category key detailed data are compared to obtain comparison result information.

[0074] Furthermore, when the data migration program in the memory 130 is executed by the processor, the following steps are also implemented:

[0075] Acquire data change track information by pre-generating a data plug-in, wherein the data change track information includes data domain change information, data field change information and the data table change information;

[0076] Performing data standardization processing on the data domain change information, the data field change information and the data table change information to obtain standardized change trajectory information;

[0077] The standardized change trajectory information is asynchronously stored in a preset trajectory information database.

[0078] Furthermore, when the data migration program in the memory 130 is executed by the processor, the following steps are also implemented:

[0079] Continuously acquiring the transaction log files of the source database, and parsing the transaction log files to obtain data table operation information;

[0080] The data table of the source system is continuously monitored according to the data table operation information.

[0081] Furthermore, when the data migration program in the memory 130 is executed by the processor, the following steps are also implemented:

[0082] Generate a message corresponding to the data table change information, and store the message corresponding to the data table change information in a preset distributed publish-subscribe message kafka system;

[0083] If a message corresponding to the data table change information published by the kafka system is monitored, the message corresponding to the data table change information is parsed to obtain the restored data table change information;

[0084] According to the restored data table change information, a target data field corresponding to the changed data table is determined from the plurality of data fields.

[0085] Furthermore, when the data migration program in the memory 130 is executed by the processor, the following steps are also implemented:

[0086] Analyze the hierarchical relationship of the target data field to determine the root node corresponding to the target data field;

[0087] Generate corresponding asynchronous tasks according to the root nodes corresponding to the target data fields respectively, and store the asynchronous tasks in a preset asynchronous task queue;

[0088] Based on the plurality of asynchronous tasks, data corresponding to each of the plurality of target data fields is obtained through a preset Java persistence interface, and the data corresponding to the target data fields is migrated from the source system to the target system.

[0089] This embodiment adopts the above scheme, specifically by continuously monitoring the data table of the source system, wherein the source system includes several preset data fields, and any data field includes at least one corresponding data table; if the data table of the source system is monitored to change, the corresponding data table change information is obtained; according to the data table change information, the target data field corresponding to the changed data table is determined from the several data fields; and the data in the target data field is migrated from the source system to the target system. In this embodiment, a migration method based on data fields is adopted, by continuously monitoring the changes of the data table of the source system, and migration is only performed when the data table changes. In this way, incremental migration can be achieved, which reduces the impact on the business while reducing performance overhead, and improves the efficiency and reliability of data migration.

[0090] Reference Figure 2 The first embodiment of the data migration method of the present application provides a flow chart, and the data migration method includes:

[0091] Step S10: continuously monitor the data table of the source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table.

[0092] Specifically, the current data migration method is a table-to-table method for migrating existing data. The data of each table in the source database is migrated one by one to the corresponding table in the target database. This method has high performance overhead and affects business. Specifically, it may manifest as follows: (1) High performance overhead: Each time the entire data set is re-migrated, even if only a small part of the data has changed, a lot of time and resources will be wasted, causing unnecessary pressure on database and network resources. (2) Business interruption: It usually requires downtime or a specific time window to perform table-to-table migration, which will cause business interruption, affect business continuity and availability, and cause user dissatisfaction. (3) Data inconsistency: Since the migration is one-time, data updates that occur during downtime cannot be reflected in the target system in a timely manner, resulting in data inconsistency. (4) Incremental migration problem: The table-to-table method usually does not support incremental migration and requires re-migrating all data, which increases resource overhead and risk. (4) Not suitable for heterogeneous systems: If the table structures of the source system and the target system are not exactly the same, the table-to-table method may not be able to adapt and requires additional data conversion and mapping work.

[0093] To this end, this embodiment proposes to migrate incremental data based on the data domain. The source system involved in this embodiment refers to the system that stores the original data, and the target system is the destination of data migration, that is, the system where the data is ultimately stored and used. The goal of data migration is to transfer, convert and load data from the source system to the target system for further processing, analysis or storage in the target system.

[0094] It should be noted that the data tables of the source system are continuously monitored. This process is to capture any changes in the data in real time and ensure the timeliness and accuracy of data migration. The source system is divided into multiple preset data fields, each of which contains at least one data table. This division helps to manage and monitor data more finely, so that data migration can be carried out by field rather than simply by table. Such a design can improve the efficiency and accuracy of data migration, especially when dealing with large amounts of data and complex data structures. The continuous monitoring process can be achieved through database log tracking, event triggers or other monitoring tools, which can capture data addition, modification or deletion operations in real time.

[0095] For example, in a specific implementation, the system monitors all preset data fields in the source database by configuring a database monitoring tool, such as a CDC (Change Data Capture) tool. When any data table changes are detected, the system will immediately record these changes to provide basic data for subsequent data migration and verification.

[0096] In addition, it should be noted that the source system includes several preset data fields, and any data field includes at least one corresponding data table. The data field can be understood as the classification or scope of data, which is usually divided according to business logic or function. The data fields involved in this embodiment are preset, which means that some data fields are determined in advance, and each data field includes at least one data table. For example, for insurance business, there may be data fields corresponding to customer information, data fields corresponding to policy information, data fields corresponding to claim records, etc.

[0097] In order to achieve incremental data migration, it is necessary to continuously monitor the data tables of the source system based on database triggers, transaction log files or special data change monitoring tools to capture changes in the data tables, such as inserting, updating or deleting data in the data tables.

[0098] Step S20: If a change in the data table of the source system is detected, corresponding data table change information is obtained.

[0099] It should be noted that in the embodiment of the present application, when the system monitors changes in the data table of the source system, it will obtain the corresponding data table change information, with the purpose of collecting detailed change data, including the time and type (addition, modification or deletion) of the change and the specific content of the change. This information is crucial for subsequent data migration and verification because it can ensure that only changed data is migrated, thereby improving migration efficiency and reducing unnecessary data transmission. In addition, it should be noted that the process of obtaining change information can be achieved by parsing database logs, event notifications or directly querying the database.

[0100] For example, in a specific implementation, when the system detects changes to a data table through a database monitoring tool, it captures detailed information about the changes and stores them in a message queue, such as Kafka, for subsequent processing.

[0101] Specifically, when a change is detected in the data table of the source system, such as when new data is inserted, existing data is updated, or data is deleted, the corresponding data table change information will be triggered. The content of the data table change information includes at least the table name corresponding to the data table, and may also include one or more of the table identifier, the specific change type (addition, update, deletion, etc.), the changed data record, and the timestamp of the record change. The data table change information can be used to decide how to perform data migration or other related operations.

[0102] Step S30: determining a target data field corresponding to the changed data table from the plurality of data fields according to the data table change information.

[0103] It should be noted that determining the target data field corresponding to the changed data table based on the data table change information obtained above is to map the changes in the data table to the correct data field, thereby ensuring the accuracy and completeness of data migration. In this way, the system can identify which data fields need to be updated and the specific scope of the update. The step of determining the target data field can be achieved by querying the mapping relationship between the preset data field and the data table. This mapping relationship can be statically configured or dynamically generated.

[0104] For example, in a specific implementation, the system will associate the captured data table change information with the corresponding data field according to the predefined mapping relationship between the data field and the data table, thereby determining the data field that needs to be migrated.

[0105] In addition, it should be noted that a subordinate relationship between a data table and a data field is established in advance. Usually, a data field will include one or more corresponding data tables. When data table change information is obtained, the changed data table can be determined based on the data table change information, and then the data field to which the changed data table belongs, that is, the target data field, can be determined based on the changed data table and the subordinate relationship between the data table and the data field.

[0106] Step S40: Migrate the data in the target data domain from the source system to the target system.

[0107] It should be noted that migrating the data in the target data domain from the source system to the target system is the core operation in the data migration process. Its purpose is to accurately copy the changed data in the source system to the target system to ensure data consistency and integrity. This can be achieved through batch processing or real-time synchronization, depending on business needs and system architecture. Data conversion and formatting may be required during the data migration process to ensure the compatibility and availability of the data in the target system.

[0108] For example, in a specific implementation, the system will trigger data migration operations through an asynchronous task queue based on the target data domain determined in step S30. These asynchronous tasks will obtain data from the source system through a Java persistence interface (such as JPA) and migrate it to the target system, while performing necessary data conversion and verification to ensure the accuracy and completeness of the data.

[0109] In addition, it should be noted that after determining the target data domain, data belonging to the target data domain can be extracted from the source system. In some cases, the data format of the source system may not match the target system, and data conversion is required. The data conversion process involves operations such as data format mapping, value conversion, and data cleaning to ensure that the data can be correctly accepted and understood in the target system. Once the data is extracted and converted, they will be loaded into the corresponding data domain of the target system. The loading process involves inserting, updating, or deleting data tables to ensure that the data is correctly stored in the target system. It can be understood that data migration includes the above-mentioned three stages of extraction, conversion, and loading.

[0110] Optionally, after migrating the data in the target data domain from the source system to the target system, the data before and after the migration may be compared to ensure the consistency and integrity of the data.

[0111] In a feasible implementation manner, the data migration method further includes steps A10 to A30 before step S40:

[0112] Step A10: Acquire source domain root information corresponding to the target data domain, and target domain root information corresponding to the source domain root information in the target system;

[0113] Step A20: judging whether there is a source domain root in the target data domain according to the source domain root information;

[0114] Step A30: If the source domain root does not exist in the target data domain, the target domain root in the target system is deleted according to the target domain root information.

[0115] It should be noted that, refer to Figure 3 , Figure 3 A schematic diagram of a physical deletion verification and processing flow is provided for the data migration method of the present application. Before executing data migration, the data migration method in the embodiment of the present application must first ensure data consistency between the source system and the target system. To this end, the system obtains the source domain root information corresponding to the target data domain and the corresponding target domain root information in the target system. Domain root information refers to key data with unique identification and hierarchical relationship in the data domain. They are important reference points in the data migration process. By comparing this information, the system can determine whether there is a corresponding source domain root in the target data domain. This is a key step to ensure data consistency.

[0116] In addition, it should be noted that if the target domain root corresponding to the source domain root does not exist in the target data domain, the system will delete the corresponding target domain root in the target system based on the target domain root information. The purpose of this operation is to keep the data between the source system and the target system synchronized and avoid potential problems caused by data inconsistency. The deletion operation ensures that during the data migration process, any data items that no longer exist in the source system will not appear in the target system, thereby maintaining the integrity and accuracy of the data.

[0117] For example, in a feasible implementation, the system first obtains the information of the source domain root and the target domain root through database query or configuration file, and then compares the information to determine whether the source domain root exists in the target data domain. If it is found that the corresponding source domain root is missing in the target data domain, the system will perform a deletion operation to remove the corresponding target domain root in the target system. This process ensures that the data status of the target system is consistent with the source system before data migration, laying a solid foundation for subsequent data migration.

[0118] After completing the above steps, the system will enter the core link of data migration. According to the description in the technical briefing book, the system will continue to monitor the changes in the data table of the source system. Once a change is detected, it will obtain the change information and determine the target data field corresponding to the changed data table. Subsequently, the system will migrate the data in the target data field from the source system to the target system through asynchronous tasks and Java persistence interfaces (such as JPA) based on the obtained field root information and change information. During this process, the system will also standardize the data and asynchronously store the change track information in the track information database to facilitate subsequent data monitoring and problem troubleshooting.

[0119] In addition, the system will obtain data table change information and key detailed indicators based on heartbeat monitoring events, generate visualization components to display on the front end of the data cabin, and compare key detailed data to verify the accuracy of data migration. Through this series of comprehensive measures, the system can ensure the efficiency, accuracy and completeness of data migration.

[0120] In a feasible implementation manner, step S40 further includes steps S401 to S404:

[0121] Step S401: based on a preset heartbeat monitoring event at a preset time frequency, obtaining the data table change information and key detailed indicators corresponding to the target data field;

[0122] Step S402: generating a plurality of visualization components based on the data table change information, and displaying them on a pre-set data cabin front end;

[0123] Step S403: acquiring the first type of key detail data of the target data field from the source system according to the key detail index, and acquiring the second type of key detail data of the target data field from the target system;

[0124] Step S404: Compare the first-category key detailed data with the second-category key detailed data to obtain comparison result information.

[0125] It should be noted that after completing the migration of data from the source system to the target system, the data migration method further expands the monitoring and verification mechanism to ensure the accuracy and integrity of the data. The heartbeat monitoring event is a preset monitoring mechanism that is triggered at a certain time frequency to obtain data table change information and key detailed indicators of the target data field. Key detailed indicators refer to indicators that are critical to each data field during the data migration process and can reflect the integrity and accuracy of the data. The acquisition of these indicators helps to monitor the status of data migration in real time and promptly discover and solve possible problems.

[0126] In addition, it should be noted that the system will generate visualization components based on the acquired data table change information, which are displayed on the front end of the data cabin, so that the process and results of data migration can be monitored and managed intuitively. This visualization method helps to quickly identify abnormal situations in the data migration process and take corresponding measures.

[0127] After the data migration is completed, the system will obtain the first and second key detail data from the source system and the target system respectively according to the key detail indicators. The first type of key detail data refers to the data corresponding to the key detail indicators in the source system, while the second type of key detail data refers to the corresponding data in the target system. By comparing these two types of data, the system can obtain the comparison result information, which helps to verify the accuracy of the data migration and ensure that there is no loss or error in the data during the migration process.

[0128] For example, refer to Figure 4 , Figure 4A data statistics flow diagram of a data migration process provided for the data migration method of the present application, in a feasible implementation mode, the system first obtains data table change information and key detailed indicators through heartbeat monitoring events. Then, the system generates a series of visual components, such as charts and dashboards, which are displayed at the front end of the data cabin so that the progress and status of data migration can be monitored in real time. At the same time, the system obtains data corresponding to key detailed indicators from the source system and the target system, and conducts comparative analysis. Through this comparison, the system can verify the accuracy of data migration and make corrections in time when problems are found. This whole process forms a closed-loop data migration and verification process, ensuring the efficiency and reliability of data migration. In this process, the system also standardizes the data change trajectory information and stores it asynchronously in the trajectory information database, providing detailed historical records and tracking capabilities for data migration. Such an implementation mode not only improves the accuracy of data migration, but also provides strong support for the monitoring and management of data migration.

[0129] More specifically, in the embodiment of the present application, in order to ensure the accuracy, completeness and consistency of the data, data verification is required after the data migration is completed.

[0130] First, we can obtain the key detailed indicators corresponding to the target data domain. Key detailed indicators are standards or rules used to measure data quality and accuracy. For example, for the insurance policy data domain, key detailed indicators may include: (1) Policy number consistency: Ensure that the unique identifier of each policy is correct and consistent. (2) Insurance amount accuracy: Verify whether the insurance amount is consistent with the contract to avoid incorrect policy amounts. (3) Effective date completeness: Check whether each policy has a valid effective date. (4) Business rules specified by the insurance company: Ensure that the policy meets the company's specific requirements, such as payment cycle, insurance type, etc.

[0131] Further, according to the key detail indicators, the first type of key detail data in the target data field is obtained from the source system, and the second type of key detail data in the target data field is obtained from the target system. It can be understood that the first type of key detail data is equivalent to the data before migration, and the second type of key detail data is equivalent to the data after migration.

[0132] The first type of key detail data is compared with the second type of key detail data to determine whether there are any differences or inconsistencies between them. Specifically, check whether the fields of the two sets of data match, such as comparing whether the values ​​of a customer's name, address, telephone number, etc. are the same.

[0133] If no abnormalities are found during the comparison process, the normal data comparison result information will be obtained and stored in the preset comparison record table. Similarly, if abnormalities are found during the comparison process, such as data loss, data format error, data out of range, etc., the abnormalities will be further recorded to obtain the abnormal data comparison result information, and the abnormal data comparison result information will be stored in the preset comparison record table. The comparison record table can provide reference information for subsequent review, analysis and problem solving to ensure the quality of data migration.

[0134] This embodiment specifically introduces the data verification link through the above scheme. By obtaining the first type of key detailed data and the second type of key detailed data, the accuracy and consistency of data migration can be verified. The storage of the comparison result information in the comparison record table helps to track and record the quality of data migration, discover potential problems in advance, and repair them when necessary. This helps to ensure the quality and consistency of the migrated data, reduce the risk of erroneous data entering the target system, and improve the reliability of the overall data migration.

[0135] In a feasible implementation manner, the method further includes steps B10 to B30:

[0136] Step B10: obtaining data change track information through a pre-generated data plug-in, wherein the data change track information includes data domain change information, data field change information and the data table change information;

[0137] Step B20: performing data standardization processing on the data domain change information, the data field change information and the data table change information to obtain standardized change track information;

[0138] Step B30: asynchronously storing the standardized change trajectory information in a preset trajectory information database.

[0139] It should be noted that in the incremental data migration verification solution, obtaining data change track information through pre-generated data plug-ins is a crucial step. Data change track information includes data domain change information, data field change information, and data table change information. These information together constitute a detailed history of the data migration process. Data domain refers to the logical grouping of data, and data field refers to a single attribute in a data table. By collecting this change information, the system can track every change in the data, thereby ensuring the integrity and consistency of data migration.

[0140] In addition, it should be noted that data standardization of the collected data change trajectory information is a key link in achieving data migration accuracy. Data standardization refers to the process of converting data of different formats and structures into a unified format and structure, which helps the system to more accurately compare and analyze data changes. Through data standardization, the system can eliminate data inconsistencies and ensure the accuracy and reliability of data during the migration process.

[0141] After data standardization, the system will asynchronously store the obtained standardized change trajectory information in the preset trajectory information database. This asynchronous storage mechanism allows the system to efficiently record and save the historical records of data changes without affecting the main process of data migration. The trajectory information database serves as a central repository that stores detailed information on all data changes, providing traceability and audit capabilities for data migration.

[0142] For example, refer to Figure 5 , Figure 5 A schematic diagram of the data monitoring process during the data migration process provided for the data migration method of the present application, in a feasible implementation mode, the system first monitors the data table of the source system in real time through the data plug-in, and captures changes in any data domain, data field, and data table. These change information are collected uniformly and standardized to ensure the consistency of the data format. Subsequently, the system asynchronously stores these standardized change trajectory information in the trajectory information database for historical query and data analysis. During the entire data migration process, the system uses these detailed data change trajectory information to verify the integrity and accuracy of the data to ensure the quality of data migration. In this way, the system can not only promptly discover and correct problems in data migration, but also provide a detailed historical record of data migration to provide support for future data analysis and decision-making.

[0143] This embodiment adopts the above scheme, specifically by continuously monitoring the data table of the source system, wherein the source system includes several preset data fields, and any data field includes at least one corresponding data table; if the data table of the source system is monitored to change, the corresponding data table change information is obtained; according to the data table change information, the target data field corresponding to the changed data table is determined from the several data fields; and the data in the target data field is migrated from the source system to the target system. In this embodiment, a migration method based on data fields is adopted, by continuously monitoring the changes of the data table of the source system, and migration is only performed when the data table changes. In this way, incremental migration can be achieved, which reduces the impact on the business while reducing performance overhead, and improves the efficiency and reliability of data migration.

[0144] Further, refer to Figure 6The second embodiment of the data migration method of the present application provides a flow chart based on the above Figure 2 In the embodiment shown, the source system is provided with a source database, and the source database stores data corresponding to the plurality of data fields. The “continuous monitoring of the data table of the source system” in step S10 is further refined to include:

[0145] Step S101: continuously obtain the transaction log file of the source database, and parse the transaction log file to obtain data table operation information.

[0146] Specifically, the transaction log file of the source database records all operations such as insert, update, and delete on the source database. When the data table of the data field in the source database changes, it will first be reflected in the transaction log file. In order to monitor the data table of the source system, it is necessary to continuously obtain the transaction log file of the source database, parse the transaction log file, and convert the binary or text data in the transaction log file into readable data table operation information. Among them, the data table operation information includes the table name (or table identifier) ​​of the data table involved in the operation, the operation type, the timestamp of the operation, etc.

[0147] Step S102: continuously monitor the data table of the source system according to the data table operation information.

[0148] Specifically, by comparing the data table operation information obtained this time with the data table operation information obtained last time, it can be determined whether there is a changed data table, so that continuous monitoring of the data table of the source system can be achieved.

[0149] If the data table operation information obtained this time is consistent with the data table operation information obtained last time, it is determined that there is no changed data table, and no change in the data table of the source system has been detected; if the data table operation information obtained this time is inconsistent with the data table operation information obtained last time (for example, the data table operation information obtained this time has more records), it is determined that there is a changed data table, and change in the data table of the source system has been detected.

[0150] This embodiment ensures real-time perception of data changes in the source system through the above solution, thereby achieving real-time and accuracy of data migration. By continuously monitoring the transaction log files of the source database, the details of data table operations, including insert, update, delete, etc., can be captured immediately without waiting for a specific time window, so that data migration can respond to changes more quickly, reduce data migration delays, reduce interference with business, and also improve the reliability of data migration.

[0151] Specifically, in the embodiment of the present application, the table name corresponding to the changed data table, ie, the target data table name, can also be extracted from the analysis of the data table change information.

[0152] The subordinate relationship between the preset data table and data field is actually reflected in the correspondence between the data table name and the data field number. According to the target data table name and the correspondence between the data table name and the data field number, the number of the target data field corresponding to the target data table name can be found, and then the target data field corresponding to the changed data table can be determined.

[0153] Through the above solution, this embodiment can accurately determine which data tables have changed based on the data table change information, thereby clarifying the target data field that needs to be migrated, improving the accuracy of data migration, avoiding unnecessary migration operations, and saving resources and time. At the same time, by clarifying the target data field, migration and verification can be carried out in a targeted manner, improving the efficiency and manageability of data migration. This helps to reduce the risk of business interruption and ensure the success and accuracy of data migration.

[0154] Further, refer to Figure 7 The third embodiment of the data migration method of the present application provides a flow chart based on the above Figure 2 In the embodiment shown, after acquiring corresponding data table change information if a change in the data table of the source system is detected in step S20, and before determining, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields, step S30 also includes:

[0155] Step S001, generating a message corresponding to the data table change information, and storing the message corresponding to the data table change information in a preset distributed publish-subscribe message kafka system.

[0156] Step S002: If a message corresponding to the data table change information published by the Kafka system is monitored, the message corresponding to the data table change information is parsed to obtain restored data table change information.

[0157] Further refinement of step S30 of “determining, from the plurality of data fields, a target data field corresponding to the changed data table according to the data table change information” includes:

[0158] Step S303: determining a target data field corresponding to the changed data table from the plurality of data fields according to the restored data table change information.

[0159] Specifically, in order to improve the timeliness and scalability of the data migration process, this embodiment introduces a distributed publish-subscribe message kafka system. The kafka system is designed to process real-time data streams. It adopts a publish-subscribe model, allowing multiple producers to publish messages to topics and allowing multiple consumers to subscribe to these topics to receive messages. Kafka has the characteristics of high throughput, scalability, fault tolerance and persistence. It is suitable for applications such as large-scale data processing, log collection, event-driven architecture, etc. It is widely used in various scenarios to build real-time data pipelines and realize reliable transmission and processing of data streams.

[0160] After obtaining the data table change information, data migration is not performed directly based on the data table change information. Instead, messages corresponding to the data table change information are generated and stored in the Kafka system. Storing messages corresponding to the data table change information in the Kafka system allows other components to subscribe to these messages and respond accordingly.

[0161] On the other hand, you can continue to monitor the messages published by the Kafka system (that is, subscribe to the Kafka system). If you monitor the message corresponding to the data table change information published by the Kafka system, you can further parse the message corresponding to the data table change information to obtain the restored data table change information. Then, based on the restored data table change information, you can determine the target data field corresponding to the changed data table from several data fields.

[0162] This embodiment uses the above solution to introduce the Kafka system to provide a reliable message transmission mechanism, publish and propagate data table change information in the form of messages, enhance the scalability and stability of the system, and ensure the reliable transmission of data change information. At the same time, Kafka allows multiple components to monitor and consume these messages in real time, realizing a distributed, real-time event-driven mode, so that data migration can respond to changes in the source system more efficiently and in real time. This helps to reduce the delay of data migration and improve the efficiency and reliability of data migration.

[0163] Further, refer to Figure 8 The fourth embodiment of the data migration method of the present application provides a flow chart based on the above Figure 2 In the illustrated embodiment, the step S40 of “migrating the data in the target data domain from the source system to the target system” is further refined to include:

[0164] Step S401: Analyze the hierarchical relationship of the target data field to determine the root node corresponding to the target data field.

[0165] Step S402: acquiring data corresponding to the target data field according to the root node corresponding to the target data field, and migrating the data corresponding to the target data field from the source system to the target system.

[0166] Specifically, a data domain usually has a tree structure, including a root node, an intermediate node, and a leaf node. This tree structure represents the hierarchical relationship of nodes in the data domain. The types of nodes include: (1) Root node: The root node is the top node of the data domain hierarchy. It does not depend on other nodes. The root node usually represents a data domain and is the starting point of other nodes in the data domain. (2) Intermediate node: The intermediate node is located between the root node and the leaf node. They may depend on the root node or other intermediate nodes and may contain child nodes. (3) Leaf node: The leaf node is the end node of the hierarchy. They usually no longer contain other child nodes.

[0167] By analyzing the hierarchical relationship of the target data domain, the root node corresponding to the target data domain can be determined, because the root node corresponding to the target data domain is the starting point of the entire target data domain hierarchy, which helps to organize and understand the structure and relationship of the target data domain.

[0168] Then, the data corresponding to the target data domain (i.e., the overall data) can be obtained based on the root node corresponding to the target data domain, and the data corresponding to the target data domain can be further migrated from the source system to the target system, so that data migration at the data domain level can be achieved. Compared with the traditional table-to-table stock data migration, the incremental data migration based on the data domain adopted in this embodiment has the following advantages: (1) Data migration at the data domain level is more fine-grained and accurate, which can ensure the consistency and integrity of the migrated data with the business domain, and reduce the risk of data inconsistency. (2) Data migration at the data domain level allows data to be organized according to business logic, improves the comprehensibility and maintainability of data, and helps the consistency of business processes. (3) Data migration at the data domain level is more flexible, allowing rapid adaptation when business needs change, reducing dependence on the source system. (4) By dividing data according to domains, incremental migration can be achieved, reducing performance overhead and interruptions, and improving migration efficiency.

[0169] Through the above solution, this embodiment can determine the starting point of the data migration operation, that is, the root node of the domain, by analyzing the hierarchical relationship of the target data domain. Finding the root node helps to subsequently migrate the data of the entire target data domain in a targeted manner, improves the accuracy and completeness of data migration, ensures that all relevant data can be successfully migrated, and avoids data omissions or inconsistencies.

[0170] Further, refer to Fig. 9The fifth embodiment of the data migration method of the present application provides a flow chart based on the above Figure 8 In the embodiment shown, the number of the target data fields is several, and the step S402 of "acquiring data corresponding to the target data field according to the root node corresponding to the target data field, and migrating the data corresponding to the target data field from the source system to the target system" is further refined to include:

[0171] Step S4021, generating corresponding asynchronous tasks according to the root nodes corresponding to the respective target data fields, and storing the plurality of asynchronous tasks in a preset asynchronous task queue.

[0172] Step S4022: Based on the plurality of asynchronous tasks, data corresponding to each of the plurality of target data fields is obtained through a preset Java persistence interface, and the data corresponding to the target data fields is migrated from the source system to the target system.

[0173] Specifically, the number of target data fields may be several. In order to improve the concurrency, responsiveness and fault tolerance of data migration, this embodiment introduces an asynchronous task list to manage multiple asynchronous tasks generated during the data migration process.

[0174] First, it is necessary to generate corresponding asynchronous tasks according to the root nodes corresponding to the target data fields. It can be understood that there is a one-to-one correspondence between the asynchronous tasks and the target data fields. Then, the asynchronous tasks are stored in a preset asynchronous task queue, which is a data structure used to manage and schedule the execution order of asynchronous tasks.

[0175] Furthermore, it is necessary to execute several asynchronous tasks in the asynchronous task queue, and several asynchronous tasks can be executed asynchronously. During the execution process, the root nodes corresponding to the target data fields are first found, and all the data in the target data field can be found along the root nodes. Therefore, the data corresponding to the target data fields can be obtained using the preset Java Persistence API, and the data corresponding to the target data field can be migrated from the source system to the target system. In this way, the incremental data migration based on the data field is completed.

[0176] This embodiment uses the above solution to introduce an asynchronous task queue to realize the parallel processing of asynchronous tasks in multiple target data fields. The data migration between different target data fields does not interfere with each other, which improves the overall efficiency of data migration. In addition, by obtaining data through the Java persistence interface, it can interact with different databases and data formats more flexibly, enhancing applicability and maintainability.

[0177] Further, refer to Fig.10 The sixth embodiment of the data migration method of the present application provides a flow chart.

[0178] Specifically, firstly, the transaction log files of the source database are continuously obtained, and the transaction log files are parsed to obtain data table operation information. According to the data table operation information, the data table of the source system can be continuously monitored.

[0179] If the data table of the source system is monitored to have changed, the corresponding data table change information can be further obtained. Then, a message corresponding to the data table change information is generated, the message corresponding to the data table change information is stored in the Kafka system, and the messages published by the Kafka system are continuously monitored. After the message corresponding to the data table change information is monitored, the message corresponding to the data table change information can be further parsed and restored to obtain the data table change information.

[0180] The data table change information records the table name (or data table name) of the changed data table. According to the table name of the changed data table and the predetermined correspondence between the data table name and the data field, the target data field corresponding to the changed data table can be determined. Furthermore, the root node of the target data field can be determined according to the hierarchical relationship of the target data field.

[0181] Since the root node is the starting point of all nodes in the target data domain, the entire data of the target data domain can be migrated from the source system to the target system according to the root node of the target data domain. Optionally, there may be multiple target data domains, and the migration process can be performed in an asynchronous task manner.

[0182] After the data migration is completed, the first type of key detailed data of the target data field can be obtained from the source system, and the second type of key detailed data of the target data field can be obtained from the target system. Further comparison of the first type of key detailed data and the second type of key detailed data can complete the data verification of the target data field. In this way, the consistency of the data before and after the migration can be ensured.

[0183] In this embodiment, a migration method based on data fields is adopted. By continuously monitoring the changes in the data tables of the source system, migration is performed only when the data tables change. In this way, incremental migration can be achieved, which reduces the impact on the business while reducing performance overhead, and improves the efficiency and reliability of data migration.

[0184] In addition, refer to Fig.11 The present application also provides a data migration device, which includes:

[0185] A monitoring module 111 is used to continuously monitor a data table of a source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table;

[0186] The acquisition module 112 is used to acquire corresponding data table change information if a change in the data table of the source system is detected;

[0187] A determination module 113, configured to determine, from the plurality of data fields, a target data field corresponding to the changed data table according to the data table change information;

[0188] The migration module 114 is used to migrate the data in the target data domain from the source system to the target system.

[0189] For the principle and implementation process of data migration in this embodiment, please refer to the above embodiments and will not be repeated here.

[0190] In addition, an embodiment of the present application also proposes a terminal device, which includes a memory, a processor, and a data migration program stored in the memory and executable on the processor, wherein the data migration program implements the steps of the data migration method described above when executed by the processor.

[0191] Since all the technical solutions of all the aforementioned embodiments are adopted when the data migration program is executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0192] In addition, an embodiment of the present application further proposes a computer-readable storage medium, on which a data migration program is stored. When the data migration program is executed by a processor, the steps of the data migration method described above are implemented.

[0193] Since all the technical solutions of all the aforementioned embodiments are adopted when the data migration program is executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0194] Compared with the prior art, the data migration method, apparatus, terminal device and storage medium proposed in the embodiments of the present application continuously monitor the data table of the source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table; if the data table of the source system is monitored to change, the corresponding data table change information is obtained; according to the data table change information, the target data field corresponding to the changed data table is determined from the plurality of data fields; and the data in the target data field is migrated from the source system to the target system. Based on the scheme of the present application, a migration method based on data fields is adopted, by continuously monitoring the data table changes of the source system, and migration is performed only when the data table changes. In this way, incremental migration can be achieved, which reduces the impact on the business while reducing performance overhead, and improves the efficiency and reliability of data migration.

[0195] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0196] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0197] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0198] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data migration method, characterized in that: The data migration method comprises: Continuously monitoring a data table of a source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table; If a change is detected in the data table of the source system, the corresponding data table change information is obtained; According to the data table change information, determining a target data field corresponding to the changed data table from the plurality of data fields; Migrate the data in the target data domain from the source system to the target system.

2. The data migration method according to claim 1, characterized in that: Before the step of migrating the data in the target data domain from the source system to the target system, the step further includes: Acquire source domain root information corresponding to the target data domain, and target domain root information corresponding to the source domain root information in the target system; According to the source domain root information, determining whether there is a source domain root in the target data domain; If the source domain root does not exist in the target data domain, the target domain root in the target system is deleted according to the target domain root information.

3. The data migration method according to claim 1, characterized in that: After the step of migrating the data in the target data domain from the source system to the target system, the following step further comprises: Based on the preset heartbeat monitoring events at a preset time frequency, the data table change information and the key detailed indicators corresponding to the target data field are obtained; Generate a number of visualization components based on the data table change information, and display them on a pre-set data cabin front end; According to the key detail indicators, obtaining the first type of key detail data in the target data field from the source system, and obtaining the second type of key detail data in the target data field from the target system; The first-category key detailed data and the second-category key detailed data are compared to obtain comparison result information.

4. The data migration method according to claim 1, characterized in that: The method further comprises: Acquire data change track information by pre-generating a data plug-in, wherein the data change track information includes data domain change information, data field change information and the data table change information; Performing data standardization processing on the data domain change information, the data field change information and the data table change information to obtain standardized change trajectory information; The standardized change trajectory information is asynchronously stored in a preset trajectory information database.

5. The data migration method according to claim 1, characterized in that: The source system is provided with a source database, and the source database stores data corresponding to the plurality of data fields. The step of continuously monitoring the data table of the source system includes: Continuously acquiring the transaction log files of the source database, and parsing the transaction log files to obtain data table operation information; The data table of the source system is continuously monitored according to the data table operation information.

6. The data migration method according to claim 1, characterized in that: After the step of acquiring corresponding data table change information if a change in the data table of the source system is monitored, and before the step of determining, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields, the method further includes: Generate a message corresponding to the data table change information, and store the message corresponding to the data table change information in a preset distributed publish-subscribe message kafka system; If a message corresponding to the data table change information published by the kafka system is monitored, the message corresponding to the data table change information is parsed to obtain the restored data table change information; The step of determining, from the plurality of data fields, a target data field corresponding to the changed data table according to the data table change information comprises: According to the restored data table change information, a target data field corresponding to the changed data table is determined from the plurality of data fields.

7. The data migration method according to claim 1, characterized in that: The number of the target data fields is several, and the step of migrating the data in the target data fields from the source system to the target system includes: Analyze the hierarchical relationship of the target data field to determine the root node corresponding to the target data field; Generate corresponding asynchronous tasks according to the root nodes corresponding to the target data fields respectively, and store the asynchronous tasks in a preset asynchronous task queue; Based on the plurality of asynchronous tasks, data corresponding to each of the plurality of target data fields is obtained through a preset Java persistence interface, and the data corresponding to the target data fields is migrated from the source system to the target system.

8. A data migration device, characterized in that: The data migration device comprises: A monitoring module, used for continuously monitoring a data table of a source system, wherein the source system includes a plurality of preset data fields, and any data field includes at least one corresponding data table; An acquisition module, configured to acquire corresponding data table change information if a change in the data table of the source system is detected; A determination module, configured to determine, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields; A migration module is used to migrate the data in the target data domain from the source system to the target system.

9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a data migration program stored in the memory and executable on the processor. When the data migration program is executed by the processor, the steps of the data migration method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data migration program, and when the data migration program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 7 are implemented.

Images