Heterogeneous database data migration monitoring method, device and system and medium

Abstract

The invention provides a heterogeneous database data migration monitoring method, device and system and a medium, and the method comprises the steps: detecting a migration task log, when an abnormal log is detected, analyzing the abnormal log, sending an abnormal signal to a system snapshot module, and outputting a log analysis result; the data analysis module is used for detecting data information, analyzing abnormal data and sending an abnormal signal to the system snapshot module when the abnormal data is detected, and outputting a data analysis result; when the system snapshot module receives the abnormal signal, a system state snapshot is generated, stored and backed up, and a system snapshot result is output; and generating an analysis report according to the log analysis result, the system snapshot result and the data analysis result, and outputting early warning information. According to the method, real-time anomaly detection and analysis of migration logs and data information are realized, key information for triggering an abnormal task can be retained in real time, an analysis report can be generated, the reliability of data migration is improved, and diagnosis and recovery capabilities are provided under the abnormal condition.

Description

Technical Field

[0001] The present invention relates to the field of data monitoring technology, and in particular to a heterogeneous database data migration monitoring method, device, system and medium. Background Art

[0002] Existing technologies can already issue alarms when data migration anomalies occur, but this process still requires manual intervention. When an anomaly occurs, operation and maintenance personnel need to delve into log files to analyze the cause of the anomaly. In addition, locating the problem often requires reference to information from other related systems. This is particularly tricky when dealing with long-running tasks or a large number of tasks in automated regression testing. Since it is impossible to obtain task exception information in a timely manner, it is not only difficult to locate those difficult and complicated problems that occasionally occur in specific scenarios, but it is also not conducive to the continuous optimization and stable operation of the system. Therefore, there is an urgent need for a more efficient and intelligent exception handling mechanism to achieve rapid discovery, accurate diagnosis and automatic repair of problems, thereby improving overall operation and maintenance efficiency and system stability. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a heterogeneous database data migration monitoring method, which realizes real-time anomaly detection and analysis of migration logs and data information, can retain key information that triggers abnormal tasks in real time, and generate analysis reports, thereby improving the reliability of data migration and providing diagnosis and recovery capabilities in the event of abnormal situations.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A first aspect of the present invention provides a method for monitoring data migration of a heterogeneous database, comprising the following steps:

[0006] Perform anomaly detection on the migration task log. When an abnormal log is detected, analyze the abnormal log and send an abnormal signal to the system snapshot module, and output the log analysis result.

[0007] Perform anomaly detection on data information. When abnormal data is detected, analyze the abnormal data and send an abnormal signal to the system snapshot module to output the data analysis results;

[0008] When the system snapshot module receives an abnormal signal, it generates a system status snapshot and performs storage backup, and outputs the system snapshot result;

[0009] An analysis report is generated based on the log analysis results, system snapshot results and data analysis results, and warning information is output.

[0010] In one embodiment, the abnormality detection is performed on the migration task log, and when an abnormal log is detected, the abnormal log is analyzed and an abnormal signal is sent to the system snapshot module, and the log analysis result is output, including:

[0011] Obtain historical exception logs and formulate preset exception rules based on the exception logs;

[0012] Get the migration task log generated when migrating data;

[0013] Detecting whether there is an abnormal log in the migration task log according to the preset abnormal rule;

[0014] When the migration log matches the preset exception rule, the migration log is considered to be an exception log;

[0015] The abnormal log is parsed and a log abnormality signal is sent to the system snapshot module to generate a log analysis result.

[0016] In one embodiment, parsing the abnormal log and sending the abnormal log signal to the system snapshot module to generate the log analysis result includes:

[0017] Preprocessing the abnormal log, analyzing the preprocessed abnormal log, and generating log analysis information;

[0018] Extracting abnormal key information from the log analysis information;

[0019] The abnormal log and abnormal key information are summarized to generate a log analysis result.

[0020] In one embodiment, the abnormality detection is performed on the data information, and when abnormal data is detected, the abnormal data is analyzed and an abnormal signal is sent to the system snapshot module, and the data analysis result is output, including:

[0021] Obtain the data information generated when migrating data, and check whether there is any abnormal data information;

[0022] When abnormal data information is detected in the data information, a data information abnormality signal is sent to the system snapshot module and a data analysis result is output.

[0023] In one embodiment, when the system snapshot module receives an abnormal signal, it generates a system status snapshot and performs storage backup, and outputs a system snapshot result, including:

[0024] When the system snapshot module receives a log exception signal and / or a data information exception signal, a system status snapshot is generated;

[0025] The system state snapshot is stored in a backup location, and a system snapshot result is output.

[0026] In one embodiment, when detecting that there is abnormal data information in the data information, sending a data information abnormality signal to the system snapshot module and outputting a data analysis result specifically includes:

[0027] Count the amount of source data and generate a data volume threshold based on the amount of source data;

[0028] The amount of migrated data when obtaining migrated data;

[0029] Analyze the amount of migrated data to determine whether the amount of migrated data reaches a data amount threshold;

[0030] When the migration data volume does not reach the data volume threshold, a migration data volume abnormality signal is sent to the system snapshot module and a data volume analysis result is output.

[0031] In one embodiment, when detecting that there is abnormal data information in the data information, sending a data information abnormality signal to the system snapshot module and outputting a data analysis result also includes:

[0032] Get the associated data generated when migrating data;

[0033] Analyze the associated data to determine whether the associated data meets the migration index;

[0034] When the associated data does not meet the migration index, a migration index abnormal signal is sent to the system snapshot module and a migration index analysis result is output.

[0035] A heterogeneous database data migration monitoring device, the device comprising:

[0036] The log parsing module is used to detect anomalies in the migration task log. When an abnormal log is detected, the abnormal log is analyzed and an abnormal signal is sent to the system snapshot module, and the log analysis result is output;

[0037] The data analysis module is used to detect abnormalities in data information. When abnormal data is detected, the abnormal data is analyzed and an abnormal signal is sent to the system snapshot module, and the data analysis result is output;

[0038] A system snapshot module, used to generate a system status snapshot and perform storage backup when the system snapshot module receives an abnormal signal, and output the system snapshot result;

[0039] The result summary module is used to generate an analysis report based on the log analysis results, system snapshot results and data analysis results, and output warning information.

[0040] A heterogeneous database data migration monitoring system, the system comprising at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the heterogeneous database data migration monitoring method.

[0043] A non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the heterogeneous database data migration monitoring method.

[0044] Beneficial Effects : The embodiments of the present invention disclose a heterogeneous database data migration monitoring method, device, system and medium. Compared with the prior art, the present invention realizes real-time anomaly detection and analysis of migration logs and data information, can retain key information that triggers abnormal tasks in real time, and generate analysis reports, thereby improving the reliability of data migration and providing diagnosis and recovery capabilities in the event of abnormal situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0046] Figure 1 A flowchart of a heterogeneous database data migration monitoring method provided by an embodiment of the present invention;

[0047] Figure 2 Another flow chart of the heterogeneous database data migration monitoring method provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the functional modules of a heterogeneous database data migration monitoring device provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the hardware structure of a heterogeneous database data migration monitoring system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] See also Figure 1, Figure 1 This is a flow chart of an embodiment of a heterogeneous database data migration monitoring method provided by the present invention. The method specifically includes the following steps:

[0052] S100, performing anomaly detection on the migration task log, and when an abnormal log is detected, analyzing the abnormal log and sending an abnormal signal to the system snapshot module, and outputting the log analysis result;

[0053] S200, performing anomaly detection on the data information, and when abnormal data is detected, analyzing the abnormal data and sending an abnormal signal to the system snapshot module, and outputting the data analysis result;

[0054] S300, when the system snapshot module receives an abnormal signal, generates a system status snapshot and performs storage backup, and outputs a system snapshot result;

[0055] S400, generating an analysis report according to the log analysis results, system snapshot results and data analysis results, and outputting warning information.

[0056] In this embodiment, during the data migration process, the system continuously monitors the generated migration task logs to promptly detect any abnormal log entries. When an abnormal log is detected, the system will conduct an in-depth analysis to clarify the nature of the abnormality and its potential causes. After the analysis is completed, the system will output detailed log analysis results and send an abnormality alert to the system snapshot module. At the same time, the system is also monitoring the data during the migration process to identify possible abnormal data information. If abnormal data information is detected, the system will analyze it to find the root cause of the inconsistent or incorrect data information. The analysis results will be output, and an abnormal signal will also be sent to the system snapshot module.

[0057] The system snapshot module is responsible for receiving abnormal signals from the log analysis and data analysis modules. Once an abnormal signal is received, the module will immediately create a snapshot of the system status, which contains key information and system configuration during the migration process. These system snapshots will be stored and backed up for subsequent problem diagnosis and system recovery.

[0058] The system will automatically generate a comprehensive analysis report based on the log analysis results, system snapshots, and data analysis results. The report will describe the abnormalities in the migration task in detail and provide the data migration team with the necessary information to quickly respond to the abnormalities and take appropriate corrective measures. Through this process, the present invention ensures real-time abnormal monitoring and analysis of log and data information during the data migration process, and can save key information in real time under abnormal circumstances. This not only improves the reliability of data migration, but also provides a quick means of diagnosis and recovery for potential problems.

[0059] In one embodiment, the abnormality detection is performed on the migration task log, and when an abnormal log is detected, the abnormal log is analyzed and an abnormal signal is sent to the system snapshot module, and the log analysis result is output, including:

[0060] Obtain historical exception logs and formulate preset exception rules based on the exception logs;

[0061] Get the migration task log generated when migrating data;

[0062] Detecting whether there is an abnormal log in the migration task log according to the preset abnormal rule;

[0063] When the migration log matches the preset exception rule, the migration log is considered to be an exception log;

[0064] The abnormal log is parsed and a log abnormality signal is sent to the system snapshot module to generate a log analysis result.

[0065] In this embodiment, during the data migration process, the system will generate a large amount of log information, which records the execution details of the migration task in detail, such as data transmission, processing steps, system status changes, etc. In order to monitor this information, the monitoring system needs to first obtain the migration task log, which is usually done through a log collector or agent, which can read or check the log file in real time and transmit the log data to the monitoring system. The monitoring system will analyze the collected logs in real time to identify whether there are abnormal logs. Abnormal logs may contain error messages, warnings, system abnormalities, performance bottlenecks, etc., which are signs of problems that may be encountered during the migration process. Detecting abnormal logs usually relies on predefined rules, pattern matching, or machine learning models, which help the system automatically identify abnormal entries.

[0066] When anomaly logs are detected, the monitoring system will parse these logs to determine the specific content and potential causes of the anomaly. The parsing process may include in-depth analysis of log entries to extract key information such as timestamps, error codes, affected components, etc. The system may also classify the anomalies to facilitate subsequent troubleshooting and problem solving. After parsing the anomaly log, the monitoring system will send a log exception signal to the system snapshot module. This signal serves as a notification to tell the system snapshot module that an anomaly has occurred in the migration task and that immediate action is required. After receiving the signal, the system snapshot module will perform the task of creating a system status snapshot.

[0067] After the monitoring system has completed parsing the exception log, it will generate log analysis results. These results record in detail the content of the exception log, the key information obtained from the analysis, the possible causes, and the recommended solutions. These log analysis results are crucial to understanding the problems that occurred during the migration task, and they provide the data migration team with the necessary information to quickly respond to and solve the problems.

[0068] If no abnormal logs are found, the system will start the data volume analysis process. This process involves statistics and verification of the amount of migrated data, including indicators such as the number of records and data size. The system may perform a series of data consistency checks, such as comparing the number of records in the source and target databases, or verifying the integrity and accuracy of the data.

[0069] After the data volume analysis is completed, the system will output the analysis results, which include statistics of data migration, the achievement of performance indicators, and any observed anomalies or deviations. These output results help monitor the progress of the migration task and provide the migration team with a comprehensive perspective of the data migration. If the data volume analysis results show that the migration task is proceeding as planned and all data has been successfully migrated, the migration team can continue with subsequent tasks or perform post-migration verification work. If the analysis results indicate that there are problems, such as the data volume is not in line with expectations, the system may initiate further investigation or re-execute the migration task.

[0070] Through this process, the monitoring system can promptly detect and respond to abnormal situations in the migration task. By generating and analyzing abnormal logs and working in conjunction with the system snapshot module, the monitoring system improves the reliability and security of data migration, ensures the smooth progress of the migration task, and can take prompt action when problems arise.

[0071] In one embodiment, parsing the abnormal log and sending the abnormal log signal to the system snapshot module to generate the log analysis result includes:

[0072] Preprocessing the abnormal log, analyzing the preprocessed abnormal log, and generating log analysis information;

[0073] Extracting abnormal key information from the log analysis information;

[0074] The abnormal log and abnormal key information are summarized to generate a log analysis result.

[0075] In this embodiment, in the preprocessing stage, the log data is cleaned and irrelevant information, such as blank lines, comments, and formatting characters, is removed to keep the log content pure. Format unification: Considering that different systems may generate logs in different formats, these logs need to be converted into a unified format for subsequent analysis. Timestamp correction: Ensure that the timestamps of all log entries are accurate, and perform time zone conversion for logs with time zone differences. Log classification: Classify according to log type (error, warning, information, debugging, etc.) to facilitate subsequent analysis by category.

[0076] Then, by applying regular expressions, keyword matching and other techniques, we can identify abnormal patterns in the logs, such as error codes and exception stacks. We can also perform statistical analysis on abnormal events, including frequency, time distribution and impact range, and analyze the changing trend of abnormal events over time to identify the nature of the problem, such as periodicity, suddenness or persistence.

[0077] Finally, identify and extract information that is critical to analyzing the cause of the anomaly, such as the error type, description, time of occurrence, and affected modules or services. Use text mining techniques, such as natural language processing (NLP) and machine learning models, to automatically extract key information and analyze the context of the anomaly log to identify the causes and consequences of the anomaly. Subsequently, design an information summary template that includes the log source, anomaly type, key information, possible impact range, and recommended troubleshooting directions, integrate the preprocessed logs and key information, form a structured analysis report, and present it in the form of a chart, list, or document. The analysis results can be output through email, system notifications, online documents, etc.

[0078] Through these steps, valuable information can be efficiently extracted from a large number of exception logs, helping operation and maintenance personnel to quickly locate the root cause of the problem and take corresponding measures to solve the problem.

[0079] In one embodiment, the abnormality detection is performed on the data information, and when abnormal data is detected, the abnormal data is analyzed and an abnormal signal is sent to the system snapshot module, and the data analysis result is output, including:

[0080] Obtain the data information generated when migrating data, and check whether there is any abnormal data information;

[0081] When abnormal data information is detected in the data information, a data information abnormality signal is sent to the system snapshot module and a data analysis result is output.

[0082] In this embodiment, during the data migration process, the system will generate and process a large amount of data information, including the actual migrated data content, data structure, data integrity check value, etc. In order to perform subsequent anomaly detection, the monitoring system needs to collect this data information. The data acquisition method may involve directly accessing the database, reading data files, or using the interface provided by the data migration tool.

[0083] The monitoring system will analyze the collected data information in real time to identify whether there is any abnormal data information. Once the monitoring system detects abnormal data, it will immediately send an abnormal signal to the system snapshot module. This signal serves as a trigger to notify the system snapshot module that an abnormal situation related to data has occurred during the migration process and that immediate measures need to be taken to save the current system status for subsequent problem diagnosis and analysis.

[0084] While detecting anomalies and sending signals, the monitoring system generates data analysis results. These results record in detail the specific content, location, possible impact range, and possible causes of abnormal data. These data analysis results are crucial to understanding data problems that arise during the migration process and provide the data migration team with the necessary information to quickly respond and resolve problems.

[0085] Based on the results of data analysis, the data migration team can implement appropriate measures, such as re-migrating problematic data, fixing data problems, or adjusting migration strategies to prevent future problems. If abnormal data affects business operations, it may also be necessary to notify the relevant business team and provide a temporary solution.

[0086] Through this process, the monitoring system can detect and respond to anomalies in data information in real time during the data migration process. By generating and analyzing abnormal data and working in conjunction with the system snapshot module, the monitoring system enhances the reliability and security of data migration, ensures the integrity and accuracy of the migrated data, and can take prompt action when problems arise.

[0087] In one embodiment, when the system snapshot module receives an abnormal signal, it generates a system status snapshot and performs storage backup, and outputs a system snapshot result, including:

[0088] When the system snapshot module receives a log exception signal and / or a data information exception signal, a system status snapshot is generated;

[0089] The system state snapshot is stored in a backup location, and a system snapshot result is output.

[0090] In this embodiment, the system snapshot module is a key function in the data migration monitoring system, which is responsible for monitoring the abnormal signals from the log analysis module and the data analysis module. Whether the log analysis module detects abnormal logs and sends log abnormal signals, or the data analysis module detects abnormal data and sends data information abnormal signals, the system snapshot module will receive these signals.

[0091] Once an abnormal signal is captured, the system snapshot module will quickly create a system status snapshot. The system status snapshot is an instant record of the current system configuration and operating status, which includes but is not limited to the following: system configuration parameters, memory usage, CPU load, network status and connection information, database transaction logs, running processes and service status, and the status and configuration of data migration tools. The purpose of the system snapshot is to capture the entire system environment when the abnormality occurs, so as to facilitate subsequent problem diagnosis and analysis.

[0092] The created system state snapshot needs to be securely stored in a designated backup location, which can be a local storage device (such as a hard disk or solid-state drive), network attached storage (NAS), storage area network (SAN) or cloud storage service. The purpose of storing snapshots is to ensure that the information can be quickly accessed when needed and to prevent data loss or corruption.

[0093] After generating and storing a system status snapshot, the system snapshot module will output the system snapshot results. These results usually contain snapshot identification information, generation time, storage location, and guidance on how to access and use these snapshots. These output results are very important reference materials for the data migration team because they provide the system status when the exception occurs, which helps to quickly locate the problem and restore the system.

[0094] Using the system state snapshot, the data migration team can analyze the data in the snapshot to determine the cause of the anomaly, assess the scope of impact, and develop appropriate repair measures. If necessary, the team can use the system snapshot to restore the system to the state before the anomaly occurred, or use the information in the snapshot to optimize the migration strategy and prevent future anomalies.

[0095] Through this process, the system snapshot module provides an effective recording and backup mechanism for abnormal situations that may occur during data migration. This not only helps to quickly respond to and solve problems during the migration process, but also improves the overall reliability and security of data migration.

[0096] In one embodiment, when detecting that there is abnormal data information in the data information, sending a data information abnormality signal to the system snapshot module and outputting a data analysis result specifically includes:

[0097] Count the amount of source data and generate a data volume threshold based on the amount of source data;

[0098] The amount of migrated data when obtaining migrated data;

[0099] Analyze the amount of migrated data to determine whether the amount of migrated data reaches a data amount threshold;

[0100] When the migration data volume does not reach the data volume threshold, a migration data volume abnormality signal is sent to the system snapshot module and a data volume analysis result is output.

[0101] In this embodiment, before the data migration begins, the system counts the amount of data in the source database. This may include the number of records, the size of the data table, the total storage capacity of the database, etc. Based on the statistically obtained amount of source data, the system generates a data volume threshold. This threshold is usually a percentage or a fixed value based on the amount of source data, which is used to evaluate whether the migration task is successful. The setting of the data volume threshold may take into account normal data loss or changes that may occur during the data migration process, for example, a reduction in the number of records due to data cleaning or conversion.

[0102] During the data migration process, the system will generate detailed information about the amount of data being migrated and related performance data. The amount of data being migrated refers to the total amount of data actually transferred during the migration process, while related performance data may include data transfer rate, frequency of migration operations (such as number of queries per second, QPS), latency of data migration, etc. This information can be collected through the monitoring interface of the data migration tool, log files, database queries, or a dedicated monitoring system.

[0103] The system analyzes the amount of data collected for migration to determine whether it has reached the preset data volume threshold. The data volume threshold is a pre-set indicator used to measure the scale and performance of the migration task. If the amount of data migrated does not reach the threshold, this may indicate data loss, inefficient transmission, or other problems during the migration process. When the analysis results show that the amount of data migrated does not reach the threshold, the system sends a migration data volume exception signal to the system snapshot module to notify the system snapshot module to capture the current system status for subsequent problem diagnosis and analysis. The system outputs the results of the data volume analysis, including the actual amount of data migrated, comparison with the threshold, possible cause analysis, and recommended countermeasures.

[0104] The system can promptly detect and respond to situations where the amount of data to be migrated does not meet expectations during the data migration process. By generating and analyzing abnormal signals and working in conjunction with the system snapshot module, the reliability and security of data migration are improved. This helps ensure the integrity of the migration task and enables quick action when problems arise.

[0105] In one embodiment, when detecting that there is abnormal data information in the data information, sending a data information abnormality signal to the system snapshot module and outputting a data analysis result also includes:

[0106] Get the associated data generated when migrating data;

[0107] Analyze the associated data to determine whether the associated data meets the migration index;

[0108] When the associated data does not meet the migration index, a migration index abnormal signal is sent to the system snapshot module and a migration index analysis result is output.

[0109] In this embodiment, during the data migration process, in addition to the amount of migrated data itself, some data directly related to the migration process will be generated, which are called associated data. Associated data may include the frequency of migration operations (such as the number of records or data volume migrated per second), traffic during the migration process, response time or delay of migration operations, system resource usage (such as CPU and memory usage), and the number of occurrences of any errors or abnormal events. Associated data can be obtained through the monitoring interface of the data migration tool, log files, database queries, or a dedicated monitoring system. The system will analyze the acquired associated data to determine whether these data meet the predetermined migration indicators. Migration indicators are a set of predefined performance and quality standards used to evaluate the efficiency and effectiveness of the migration process.

[0110] Indicators include:

[0111] Whether the migration rate (such as the number of records or data volume migrated per second) reaches the predetermined target.

[0112] Whether the response time or latency of the migration operation is within acceptable limits.

[0113] Check whether the usage of system resources is within a reasonable range and is not too high or too low.

[0114] Whether the number of errors or abnormal events is within the control threshold.

[0115] The analysis process may involve comparing actual correlation data with predetermined migration metrics, using statistical methods or machine learning models to identify anomalous patterns, and evaluating the overall performance of the migration process.

[0116] If the analysis results show that the associated data does not meet the predetermined migration indicators, this may indicate performance problems or quality defects in the migration process. In this case, the system will send a migration indicator abnormality signal to the system snapshot module. This signal is used to notify the system snapshot module to capture the current system status for subsequent problem diagnosis and analysis.

[0117] Through this process, the system can monitor and respond to abnormal data volume and performance indicators in real time during data migration. By generating and analyzing abnormal signals and working in conjunction with the system snapshot module, the system improves the reliability and security of data migration, ensures the smooth progress of migration tasks, and can take prompt action when problems arise.

[0118] See also Figure 2 , Figure 2 This is a flow chart of another embodiment of the heterogeneous database data migration monitoring method provided by the present invention, and the method specifically includes the following steps:

[0119] (1) Detection log;

[0120] (2) Determine whether there is an abnormal log. If there is an abnormal log, proceed to step (3); if there is no abnormal log, proceed to step (4);

[0121] (3) Analyze exception logs;

[0122] (4) Analysis of the amount of migrated data;

[0123] (5) Migration index analysis;

[0124] (6) Determine whether the migration index is met. If not, proceed to step (7);

[0125] (7) Generate a system status snapshot;

[0126] (8) Storage system status snapshot;

[0127] (9) The results are summarized and a report is generated.

[0128] In the data migration process of the present invention, the system continuously monitors the logs generated by the migration task. When an abnormal log is detected, the system outputs the analysis result, notifies the snapshot module to generate a state snapshot, and backs it up for recovery. The system also monitors the data information in the migration process to detect any abnormal data. If abnormal data is found, the system will analyze it to identify the root cause of data inconsistency or error. The data analysis result will be output, and an abnormal signal will also be sent to the system snapshot module. The system snapshot module is responsible for monitoring the abnormal signals from the log analysis and data analysis modules. When the abnormal signal is received, the system snapshot module will immediately generate a snapshot of the system state to ensure that it can be used for problem diagnosis and system recovery when needed. The system will automatically generate a comprehensive analysis report based on the log analysis results, the system snapshot results and the data analysis results. Through this process, the present invention realizes real-time abnormal detection and analysis of migration logs and migration data information during data migration, and can trigger the retention of key information of abnormal tasks in real time. This not only improves the reliability of data migration, but also provides rapid diagnosis and recovery capabilities for problems that may occur during the migration process.

[0129] A heterogeneous database data migration monitoring device, the device comprising:

[0130] The log analysis module 11 is used to detect anomalies in the migration task log. When an abnormal log is detected, the abnormal log is analyzed and an abnormal signal is sent to the system snapshot module, and the log analysis result is output;

[0131] The data analysis module 12 is used to detect abnormalities in data information. When abnormal data is detected, the abnormal data is analyzed and an abnormal signal is sent to the system snapshot module, and the data analysis result is output;

[0132] The system snapshot module 13 is used to generate a system status snapshot and perform storage backup when the system snapshot module receives an abnormal signal, and output the system snapshot result;

[0133] The result summary module 14 is used to generate an analysis report based on the log analysis results, system snapshot results and data analysis results.

[0134] like Figure 3 As shown, the log parsing module 11, the data analysis module 12, the system snapshot module 13 and the result summary module 14 are connected in sequence. The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for the execution process of the heterogeneous database data migration monitoring method than the program. For the specific implementation method of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0135] In one embodiment, the log parsing module 11 includes:

[0136] A log detection unit, used to obtain a migration task log generated when migrating data, and detect whether there is an abnormal log in the migration task log;

[0137] The log parsing unit is used to parse the abnormal log and send a log abnormality signal to the system snapshot module to generate a log analysis result when detecting that there is an abnormal log in the migration task log.

[0138] In one embodiment, the log parsing module 11 further includes:

[0139] The migration parsing unit is used to parse the migration data volume when detecting that no abnormal log exists in the migration task log.

[0140] In one embodiment, the data analysis module 12 includes:

[0141] A data information detection unit, used to obtain data information generated when migrating data, and detect whether the data information contains abnormal data information;

[0142] The data information analysis unit sends a data information abnormality signal to the system snapshot module and outputs a data analysis result when detecting that there is abnormal data information in the data information.

[0143] In one embodiment, the system snapshot module 13 includes:

[0144] A system status snapshot unit, configured to generate a system status snapshot when the system snapshot module receives a log exception signal and / or a data information exception signal;

[0145] The system state storage unit is used to store the system state snapshot in a backup location and output the system snapshot result.

[0146] In one embodiment, the data information detection unit specifically includes:

[0147] The amount of migration data and associated data generated when obtaining migration data;

[0148] Analyze the migration data volume to determine whether the migration data volume reaches a data volume threshold, and when the migration data volume does not reach the data volume threshold, send a migration data volume abnormality signal to a system snapshot module and output a data volume analysis result;

[0149] The associated data is analyzed to determine whether the associated data meets the migration index. When the associated data does not meet the migration index, a migration index abnormality signal is sent to the system snapshot module and a migration index analysis result is output.

[0150] In one embodiment, the result aggregation module 14 includes:

[0151] The analysis report generating unit is used to generate an analysis report according to the log analysis results, system snapshot results, data volume analysis results and migration indicator analysis results.

[0152] Another embodiment of the present invention provides a heterogeneous database data migration monitoring system, the system comprising at least one processor; and

[0153] a memory communicatively connected to the at least one processor; wherein,

[0154] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the heterogeneous database data migration monitoring method.

[0155] One or more processors 110 and memory 120, Figure 4 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0156] The processor 110 is used to complete various control logics of the control system 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0157] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the heterogeneous database data migration monitoring method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the control system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the heterogeneous database data migration monitoring method in the above method embodiment.

[0158] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the control system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the control system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] One or more units are stored in the memory 120, and when executed by one or more processors 110, the heterogeneous database data migration monitoring method in any of the above method embodiments is executed, for example, the above described Figure 1 The method comprises steps S100 to S400.

[0160] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 The method comprises steps S100 to S400.

[0161] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0162] In summary, the present invention discloses a heterogeneous database data migration monitoring method, device, system and medium, including: performing anomaly detection on the migration task log, when an abnormal log is detected, analyzing the abnormal log and sending an abnormal signal to the system snapshot module, and outputting the log analysis result; performing anomaly detection on the data information, when abnormal data is detected, analyzing the abnormal data and sending an abnormal signal to the system snapshot module, and outputting the data analysis result; when the system snapshot module receives the abnormal signal, generating a system state snapshot for storage backup, and outputting the system snapshot result; generating an analysis report based on the log analysis result, the system snapshot result and the data analysis result. The present invention realizes real-time anomaly detection and analysis of migration logs and data information, can retain the key information that triggers abnormal tasks in real time, and generate an analysis report, improve the reliability of data migration, and provide diagnosis and recovery capabilities in abnormal situations.

[0163] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0164] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for monitoring data migration of heterogeneous databases, characterized in that: The steps include: Perform anomaly detection on the migration task log. When an abnormal log is detected, analyze the abnormal log and send an abnormal signal to the system snapshot module, and output the log analysis result. Perform anomaly detection on data information. When abnormal data is detected, analyze the abnormal data and send an abnormal signal to the system snapshot module to output the data analysis results; When the system snapshot module receives an abnormal signal, it generates a system status snapshot and performs storage backup, and outputs the system snapshot result; An analysis report is generated based on the log analysis results, system snapshot results and data analysis results, and warning information is output.

2. The heterogeneous database data migration monitoring method according to claim 1 is characterized in that: The abnormality detection is performed on the migration task log. When an abnormal log is detected, the abnormal log is analyzed and an abnormal signal is sent to the system snapshot module, and the log analysis result is output, including: Obtain historical exception logs and formulate preset exception rules based on the exception logs; Get the migration task log generated when migrating data; Detecting whether there is an abnormal log in the migration task log according to the preset abnormal rule; When the migration log matches the preset exception rule, the migration log is considered to be an exception log; The abnormal log is parsed and a log abnormality signal is sent to the system snapshot module to generate a log analysis result.

3. The heterogeneous database data migration monitoring method according to claim 2 is characterized in that: The abnormal log is parsed and a log abnormality signal is sent to the system snapshot module to generate a log analysis result, including: Preprocessing the abnormal log, analyzing the preprocessed abnormal log, and generating log analysis information; Extracting abnormal key information from the log analysis information; The abnormal log and abnormal key information are summarized to generate a log analysis result.

4. The heterogeneous database data migration monitoring method according to claim 1 is characterized in that: The data information is detected for abnormality, and when abnormal data is detected, the abnormal data is analyzed and an abnormal signal is sent to the system snapshot module, and the data analysis result is output, including: Obtain the data information generated when migrating data, and check whether there is any abnormal data information; When abnormal data information is detected in the data information, a data information abnormality signal is sent to the system snapshot module and a data analysis result is output.

5. The heterogeneous database data migration monitoring method according to claim 2 or 4 is characterized in that: When the system snapshot module receives an abnormal signal, it generates a system status snapshot and performs storage backup, and outputs a system snapshot result, including: When the system snapshot module receives a log exception signal and / or a data information exception signal, a system status snapshot is generated; The system state snapshot is stored in a backup location, and a system snapshot result is output.

6. The heterogeneous database data migration monitoring method according to claim 4 is characterized in that: When detecting that there is abnormal data information in the data information, sending a data information abnormality signal to the system snapshot module and outputting a data analysis result specifically includes: Count the amount of source data and generate a data volume threshold based on the amount of source data; The amount of migrated data when obtaining migrated data; Analyze the amount of migrated data to determine whether the amount of migrated data reaches a data amount threshold; When the migration data volume does not reach the data volume threshold, a migration data volume abnormality signal is sent to the system snapshot module and a data volume analysis result is output.

7. The heterogeneous database data migration monitoring method according to claim 4 is characterized in that: When detecting that there is abnormal data information in the data information, sending a data information abnormality signal to the system snapshot module and outputting a data analysis result, further comprising: Get the associated data generated when migrating data; Analyze the associated data to determine whether the associated data meets the migration index; When the associated data does not meet the migration index, a migration index abnormal signal is sent to the system snapshot module and a migration index analysis result is output.

8. A heterogeneous database data migration monitoring device, characterized in that: The device comprises: The log parsing module is used to detect anomalies in the migration task log. When an abnormal log is detected, the abnormal log is analyzed and an abnormal signal is sent to the system snapshot module, and the log analysis result is output; The data analysis module is used to detect abnormalities in data information. When abnormal data is detected, the abnormal data is analyzed and an abnormal signal is sent to the system snapshot module, and the data analysis result is output; A system snapshot module, used to generate a system status snapshot and perform storage backup when the system snapshot module receives an abnormal signal, and output the system snapshot result; The result summary module is used to generate an analysis report based on the log analysis results, system snapshot results and data analysis results, and output warning information.

9. A heterogeneous database data migration monitoring system, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the heterogeneous database data migration monitoring method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the heterogeneous database data migration monitoring method according to any one of claims 1 to 7.

Images