Method and device for monitoring data logic migration process

Abstract

The invention relates to the technical field of data visualization and data engineering, and particularly provides a method and device for monitoring a data logic migration process, and the method comprises the following steps: S1, a data source access module manages migration source system and target system database information; s2, selecting one or more business tables in a migration source system and a target system by a progress object selection tool, and calculating a migration process progress based on the one or more business tables; s3, the data extraction script tool writes data change and abnormal logs of the data migration comparison object into a record table; s4, the visual configuration module configures an instrument board and a report, and displays the progress and the state of the migration process; s5, actively informing a data migration operator when a migration exception occurs in an exception informing module; and S6, the data consistency checking module checks the consistency and integrity of the migrated data. Compared with the prior art, the method has the advantages that the tool repeated development cost caused by database type differences of enterprises can be reduced, and the reuse rate of migration tools is greatly increased.

Description

Technical Field

[0001] This invention relates to the fields of data visualization and data engineering technology, specifically providing a method and apparatus for monitoring the data logical migration process. Background Art

[0002] Data migration is a key task in information system construction, especially when replacing existing information systems, where there is a large demand for migrating data from old systems to new systems.

[0003] However, traditional system migration processes lack a visual interface, making it impossible for users to intuitively view migration progress and anomalies. Monitoring of the migration process is insufficient. Lack of visualization, reliance on manual monitoring for anomalies, and inadequate guarantees of post-migration data consistency and integrity are current pain points in data migration. Summary of the Invention

[0004] This invention addresses the shortcomings of the prior art by providing a highly practical method for monitoring the data logical migration process.

[0005] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable device for monitoring the data logic migration process.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A method for monitoring the logical migration process of data includes the following steps:

[0008] S1. The data source access module manages and migrates database information of the source system and the target system.

[0009] S2. The schedule indicator selection tool selects one or more business tables in the source and target systems for migration, and calculates the migration process schedule based on this.

[0010] S3. The data extraction script tool writes the data migration comparison target's data changes and anomaly logs into the record table;

[0011] S4, the visual configuration module configures dashboards and reports to display the migration process progress and status;

[0012] S5. In the exception notification module, when a migration exception occurs, the data migration operator is proactively notified.

[0013] S6, the data consistency check module checks the consistency and integrity of the migrated data.

[0014] Furthermore, in step S1, based on Datart's data source management function, the original system data source and the target system data source are defined respectively;

[0015] Database credentials are encrypted using the AES-256 algorithm and stored in the Source table of Datart;

[0016] After the data source is established, the database table structure information is automatically parsed by querying the database's INFORMATION_SCHEMA and stored in the "Progress Indicator Information Table".

[0017] Furthermore, in step S2, after the data source is connected, the source system database tables and table field information are displayed in a tree-like manner in the upper left corner of the "Progress Indicator Selection Page". Fuzzy search and regular expression filtering of table names are provided. After checking the box, one or more selected business tables are displayed on the right as progress indicators.

[0018] If there are multiple business tables, the weight setting options will be displayed. The default weight of each business table is 100 / the total number of business tables. The default weight can be modified. The weight of each business table is checked to ensure that it cannot be greater than 100 and the total weight is equal to 100. That is, the check rule is: Σ(single table weight) = 100%; single table weight ≤ 100%.

[0019] After successful verification, set the data logical migration log recording path, which is an absolute path.

[0020] Furthermore, in step S3, after the progress indicator selection is saved, the source system database information, target system database information, and selected business table information are written into the configuration file of the data extraction script tool, thereby automatically executing the data extraction script.

[0021] When the script is executed, it reads the total number of business table records, progress, and time points of the source system and target system every second and writes them into the record table.

[0022] The progress is calculated using a dynamic weighted progress algorithm, which calculates the progress based on the weight of the business table from the previous step.

[0023] The dynamic weighted progress algorithm is executed as follows:

[0024] Total weight = Σ(business table migration amount);

[0025] Completion progress = Σ(number of migrated business tables) / total weight × 100%;

[0026] At the same time, read the data logic migration log recording path set in the previous step, write the records containing "Error", "Exception" and "Waring" in the log to the exception table, and record the time when the action occurred.

[0027] Furthermore, in step S4, on the "Visualization" page of Datart, configure charts, with the data views referenced in the charts being the record table and exception table from the previous step. Then configure dashboards and reports, import the configured charts, and display the migration process progress and status, exception statistics charts, and display the distribution of exception types and time trend charts.

[0028] Furthermore, in step S5, when a new record is generated in the exception table, the exception notification module reads the operator notification channel and notification content template in the configuration file, calls the notification execution action, and sends the exception warning information to the operator.

[0029] Furthermore, in step S6, when the progress calculation in step S3 is 100, the data consistency check module is triggered to execute. The data consistency check module reads the database tables of the source system and the target system in sequence, and calculates and compares the total number of records in the source system with the same table name with the total number of records in the target system.

[0030] When both are identical, the data consistency check passes. When all business tables pass, the migration data consistency is indicated as 100%.

[0031] An apparatus for monitoring a data logical migration process includes: at least one memory and at least one processor;

[0032] The at least one memory is used to store a machine-readable program;

[0033] The at least one processor is used to call the machine-readable program to execute a method for monitoring a data logical migration process.

[0034] Compared with the prior art, the method and apparatus for monitoring the data logical migration process of the present invention have the following outstanding advantages:

[0035] This invention integrates a Datart dashboard to intuitively display migration progress and anomaly distribution, reducing the workload of manually writing monitoring reports and improving the accuracy and efficiency of migration monitoring. Through a weighted allocation mechanism, it solves the problem of progress deviation caused by differences in table size in traditional migration progress statistics.

[0036] The system collects records from the source / target systems every second and calculates progress in real time, enabling millisecond-level status feedback during the migration process. This helps migration operators respond quickly to anomalies and reduces migration risks. Multi-database adaptation mechanism: Supports JDBC connections to mainstream databases such as MySQL and Oracle, reducing the cost of redundant tool development due to database type differences and significantly improving the reusability of migration tools. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a method for monitoring the logical migration process of data. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The following is a preferred embodiment:

[0041] like Figure 1 As shown, a method for monitoring a data logical migration process in this embodiment includes the following steps:

[0042] S1. The data source access module manages and migrates database information of the source system and the target system.

[0043] Manage the migration of database information for the source and target systems. Based on Datart's data source management functionality, define the source and target system data sources separately. The data sources are mainstream databases supporting JDBC connections, including MySQL, Oracle, SQL Server, and MariaDB.

[0044] Database credentials are encrypted using the AES-256 algorithm and stored in the Source table of Datart.

[0045] After the data source is established, the database table structure information is automatically parsed by querying the database's INFORMATION_SCHEMA and stored in the "Progress Indicator Information Table".

[0046] S2. The schedule indicator selection tool selects one or more business tables in the source and target systems for migration, and calculates the migration process schedule based on this.

[0047] After the data source is connected, the "Progress Indicator Selection Page" displays the source system database tables and table field information in a tree structure in the upper left corner, providing fuzzy search and regular expression filtering of table names. After selecting a table, the right side displays one or more selected business tables as progress indicators.

[0048] If there are multiple business tables, weight setting options will be displayed. The default weight for each business table is (100 / total number of business tables). The default weight can be modified. A validation rule applies: the weight of each business table cannot exceed 100, and the total weight equals 100. The validation rules are as follows:

[0049] Σ(Single Table Weight) = 100%; Single Table Weight ≤ 100%.

[0050] After successful verification, set the data logical migration log recording path, which is an absolute path.

[0051] S3. The data extraction script tool writes the data migration comparison target's data changes and anomaly logs into the record table;

[0052] After the progress indicator is selected and saved, the source system database information, target system database information, and selected business table information are written into the configuration file of the data extraction script tool, and then the data extraction script is executed automatically.

[0053] When the script is executed, it reads the total number of business table records, progress, and time points of the source system and target system every second and writes them into the record table.

[0054] The progress is calculated using a dynamic weighted progress algorithm, which calculates the progress based on the weight of the business table from the previous step.

[0055] The dynamic weighted progress algorithm is executed as follows:

[0056] Total weight = Σ(business table migration amount);

[0057] Completion progress = Σ(number of migrated business tables) / total weight × 100%.

[0058] At the same time, read the data logic migration log recording path set in the previous step, write the records containing "Error", "Exception" and "Waring" in the log to the exception table, and record the time when the action occurred.

[0059] S4, the visual configuration module configures dashboards and reports to display the migration process progress and status;

[0060] On Datart's "Visualization" page, configure charts, with the data views referenced in the icons being the record table and exception table from the previous step. Then configure dashboards and reports, importing the configured charts to display the migration progress and status. An exception statistics chart displays the distribution of exception types and a time trend graph.

[0061] S5. In the exception notification module, when a migration exception occurs, the data migration operator is proactively notified.

[0062] When a new record is generated in the exception table, the exception notification module reads the operator's notification channel (email, mobile phone information) and notification content template (including exception time, type, and related business table information) from the configuration file, calls the notification execution action, and sends the exception warning information to the operator.

[0063] S6. The data consistency check module checks the consistency and integrity of the migrated data.

[0064] When the progress calculation in step S3 reaches 100, the data consistency check module is triggered. This module sequentially reads the database tables of the source system and the target system, and compares the total number of records with the same table name in the source system with the total number of records in the target system.

[0065] If both are identical, the data consistency check passes. If all business tables pass, the migration data consistency is considered 100%.

[0066] Based on the above method, an apparatus for monitoring a data logical migration process in this embodiment includes: at least one memory and at least one processor;

[0067] The at least one memory is used to store a machine-readable program;

[0068] The at least one processor is used to call the machine-readable program to execute a method for monitoring a data logical migration process.

[0069] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any technical solution that conforms to the above-described specific embodiments of the present invention and any appropriate changes or substitutions made by those skilled in the art should fall within the patent protection scope of the present invention.

[0070] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for monitoring the logical migration process of data, characterized in that, It has the following steps: S1. The data source access module manages and migrates database information of the source system and the target system. S2. The schedule indicator selection tool selects one or more business tables in the source and target systems for migration, and calculates the migration process schedule based on this. S3. The data extraction script tool writes the data migration comparison target's data changes and anomaly logs into the record table; S4, the visual configuration module configures dashboards and reports to display the migration process progress and status; S5. In the exception notification module, when a migration exception occurs, the data migration operator is proactively notified. S6, the data consistency check module checks the consistency and integrity of the migrated data.

2. The method for monitoring the logical migration process according to claim 1, characterized in that, In step S1, based on Datart's data source management function, the original system data source and the target system data source are defined respectively; Database credentials are encrypted using the AES-256 algorithm and stored in the Source table of Datart; After the data source is established, the database table structure information is automatically parsed by querying the database's INFORMATION_SCHEMA and stored in the "progress indicator information table".

3. The method for monitoring the logical migration process according to claim 1, characterized in that, In step S2, after the data source is connected, the source system database tables and table field information are displayed in a tree-like manner in the upper left corner of the "Progress Indicator Selection Page". Fuzzy search and regular expression filtering of table names are provided. After checking the box, one or more selected business tables are displayed on the right as progress indicators. If there are multiple business tables, the weight setting options will be displayed. The default weight of each business table is 100 / the total number of business tables. The default weight can be modified. The weight of each business table is checked to ensure that it cannot be greater than 100 and the total weight is equal to 100. That is, the check rule is: Σ(single table weight) = 100%; single table weight ≤ 100%. After successful verification, set the data logical migration log recording path, which is an absolute path.

4. The method for monitoring a data logical migration process according to claim 3, characterized in that, In step S3, after the progress indicator selection is saved, the source system database information, target system database information, and selected business table information are written into the configuration file of the data extraction script tool, and then the data extraction script is executed automatically. When the script is executed, it reads the total number of business table records, progress, and time points of the source system and target system every second and writes them into the record table. The progress is calculated using a dynamic weighted progress algorithm, which calculates the progress based on the weight of the business table from the previous step. The dynamic weighted progress algorithm is executed as follows: Total weight = Σ(business table migration amount); Completion progress = Σ(number of migrated business tables) / total weight × 100%; At the same time, read the data logic migration log recording path set in the previous step, write the records containing "Error", "Exception" and "Waring" in the log to the exception table, and record the time when the action occurred.

5. The method for monitoring a data logical migration process according to claim 4, characterized in that, In step S4, on the "Visualization" page of Datart, configure charts. The data views referenced in the charts are the record table and exception table from the previous step. Then configure dashboards and reports, import the configured charts, and display the migration process progress and status, exception statistics charts, and display the distribution of exception types and time trend charts.

6. The method for monitoring a data logical migration process according to claim 5, characterized in that, In step S5, when a new record is generated in the exception table, the exception notification module reads the operator notification channel and notification content template in the configuration file, calls the notification execution action, and sends the exception warning information to the operator.

7. The method for monitoring a data logical migration process according to claim 6, characterized in that, In step S6, when the progress calculation in step S3 is 100, the data consistency check module is triggered to execute. The data consistency check module reads the database tables of the source system and the target system in sequence, and calculates and compares the total number of records in the source system with the same table name with the total number of records in the target system. When both are identical, the data consistency check passes. When all business tables pass, the migration data consistency is indicated as 100%.

8. A device for monitoring the logical migration process of data, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to perform the method according to any one of claims 1 to 7.

Images