A fault end-to-end adaptive processing method for a change data capture system

Abstract

The application provides a fault whole-process adaptive processing method of a change data capture system, which comprises the following steps: acquiring multi-dimensional state data of the change data capture system in a running process; inputting the multi-dimensional state data into a pre-trained time sequence-global perception fault detection model, identifying and outputting fault information; executing a corresponding system recovery strategy based on a fault type and real-time abnormal parameters in the fault information; evaluating an execution effect of the system recovery strategy, and dynamically optimizing parameters of the time sequence-global perception fault detection model based on the execution effect, and simultaneously adjusting the system recovery strategy. The method provided by the application inputs the multi-dimensional state data into a specific model, realizes early prediction of system faults, executes differentiated avoidance and recovery measures for different types of faults, continuously optimizes the model and the execution measures according to the strategy execution effect, constructs a closed-loop monitoring and adaptive optimization mechanism, and significantly improves the stability and reliability of the system.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a fault adaptive handling method for the entire process of a change data capture system. Background Technology

[0002] Change Data Capture (CDC), as a core means of real-time database data synchronization, synchronizes incremental data from the source database to the target system (such as a data lake or downstream business database) in near real-time by parsing database logs (e.g., MySQL's binlog, PostgreSQL's Write-Ahead Logging, WAL) or listening for data change events. It has been widely used in critical business scenarios such as data warehouse construction, business disaster recovery, and real-time data analysis.

[0003] However, the current CDC system has the following drawbacks: it only focuses on whether the data is transmitted successfully, without coordinating the perception of multi-dimensional status data in the acquisition link, and only triggers warnings after a fault occurs or a data error is encountered, making it impossible to predict risks in advance, which increases the time of data errors or interruptions and the operation and maintenance costs; packet loss is prone to occur during data transmission, and consistency verification relies on full data coverage or comparison, which is time-consuming, computationally intensive, and difficult to achieve regular automated detection; after a fault occurs, recovery is only achieved through "task restart" or "full data synchronization". After a long pause, restarting data transmission will transmit historical data, resulting in low synchronization efficiency, and no different execution strategies are designed for different fault types. Summary of the Invention

[0004] This invention provides a fault adaptive handling method for the entire process of a change data capture system, in order to solve the technical problems of poor data transmission fault perception, low detection efficiency, passive fault triggering, single fault avoidance measures, and low recovery efficiency in existing change data capture systems.

[0005] This invention provides a fault-based adaptive handling method for the entire process of a change data capture system, comprising:

[0006] Acquire multi-dimensional status data, which includes the operating parameters and transmission parameters of each database instance in the change data capture system; The multi-dimensional state data is input into a pre-trained time-series-global awareness fault detection model to identify and output fault information; Based on the fault type and real-time abnormal parameters in the fault information, execute the corresponding system recovery strategy; The effectiveness of the system recovery strategy is evaluated, and the parameters of the time-series-global awareness fault detection model are dynamically optimized based on the effectiveness, while the system recovery strategy is adjusted.

[0007] In one embodiment of the present invention, before inputting the multi-dimensional state data into a pre-trained time-series-global perception fault detection model, the method further includes: cleaning the multi-dimensional state data to obtain intermediate data; and extracting and associating time-series features from the intermediate data to update the feature dimensions in the multi-dimensional state data.

[0008] In one embodiment of the present invention, the multi-dimensional state data is input into a pre-trained time-series-global awareness fault detection model to identify and output fault information, including: performing window sliding processing on the multi-dimensional state data based on the local feature extraction unit in the time-series-global awareness fault detection model to obtain short-term time-series data features; inputting the short-term time-series data features into the global awareness unit in the time-series-global awareness fault detection model, realizing global modeling through a cross-window attention mechanism, and performing multi-label classification processing on the global time-series data features to generate the fault information.

[0009] In one embodiment of the present invention, the construction steps of the time-series-global perception fault detection model include: acquiring training samples, the training samples including running sample data, fault pre-sample data, and fault sample data; using the training samples to jointly train the local feature extraction unit and the global perception unit to obtain sample fault information; determining the loss value between the sample fault information and the actual fault information based on a loss function; adjusting the network parameters of the local feature extraction unit and the global perception unit using the loss value until the loss value converges to a preset loss threshold to obtain the time-series-global perception fault detection model.

[0010] In one embodiment of the present invention, a corresponding system recovery strategy is executed based on the fault type and real-time abnormal parameters in the fault information, including: determining the risk level according to the fault type and the real-time abnormal parameters; matching the system recovery strategy from a preset strategy library according to the fault type, the real-time abnormal parameters and the risk level; wherein the system recovery strategy includes constructing an associated feature chain reflecting the fault propagation path based on the fault information.

[0011] In one embodiment of the present invention, evaluating the execution effect of the system recovery strategy includes: monitoring real-time abnormal parameters corresponding to the system recovery strategy based on a preset sliding window; if the real-time abnormal parameters recover from an abnormal state to within the normal indicator range or the real-time abnormal parameters show a trend of recovering to the normal indicator range, then the execution effect is determined to be effective; if the real-time abnormal parameters remain in the abnormal state or the real-time abnormal parameters show a deteriorating trend, then the execution effect is determined to be ineffective; if other related indicators different from the real-time abnormal parameters deteriorate, then the execution effect is determined to have side effects.

[0012] In one embodiment of the present invention, dynamically optimizing the parameters of the time-series-global awareness fault detection model based on the execution effect, and adjusting the system recovery strategy, includes: taking the execution effect determined to be effective as a positive sample, and taking the execution effect determined to be invalid or have side effects as a negative sample; updating the training samples based on the positive samples and the negative samples, and using the updated training samples to iteratively optimize the parameters of the time-series-global awareness fault detection model; and adjusting the execution action of the system recovery strategy corresponding to the negative sample based on the negative sample.

[0013] In one embodiment of the present invention, the change data capture system includes a source database and a target database, and the processing method further includes: obtaining the abnormal transmission time period corresponding to the fault information; performing multi-granularity consistency verification on the source database and the target database within the abnormal transmission time period, and performing data recovery operation in response to mismatched verification results.

[0014] In one embodiment of the present invention, performing multi-granularity consistency verification on the source database and the target database includes: if the amount of newly added data in the source database and the amount of newly added data in the target database are consistent, verifying the summary data of the source database and the target database to obtain a summary verification result.

[0015] In one embodiment of the present invention, in response to a mismatched verification result, a data recovery operation is performed, including: performing data recovery on the target database based on the verification result showing an inconsistent amount of newly added data; Based on the mismatched summary verification results, locate the discrepancy data, obtain the archived log corresponding to the discrepancy data, merge the archived log with the discrepancy data, and write the merged data into the target database; if the discrepancy data does not have an archived log, then synchronize the discrepancy data from the source database to the target database.

[0016] The beneficial effects of this invention are as follows: This invention proposes a fault-adaptive full-process fault handling method for a change data capture system. The method includes: acquiring multi-dimensional state data of the change data capture system during operation; inputting the multi-dimensional state data into a pre-trained time-series-full-domain awareness fault detection model to identify and output fault information; executing corresponding system recovery strategies based on the fault type and real-time abnormal parameters in the fault information; evaluating the execution effect of the system recovery strategies; dynamically optimizing the parameters of the time-series-full-domain awareness fault detection model based on the execution effect; and adjusting the system recovery strategies simultaneously. The processing method provided by this invention, by inputting real-time multi-dimensional state data into a pre-trained time-series-full-domain awareness fault detection model, can predict faults in advance and execute differentiated avoidance and recovery measures for different types of faults. It optimizes the model and avoidance measures based on the execution effect of the recovery measures, achieving closed-loop monitoring. Attached Figure Description

[0017] The accompanying drawings, incorporated in and forming part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] In the attached diagram: Figure 1 This is a flowchart illustrating the fault-based adaptive handling method for the change data capture system provided in this embodiment of the invention. Figure 2 This is a schematic diagram illustrating the sources of multi-dimensional status data in the change data capture system provided in this embodiment of the invention. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0022] MySQL is the most popular open-source relational database management system, PostgreSQL is a powerful open-source object-relational database system, and Oracle Database is a commercial relational database management system developed by Oracle Corporation.

[0023] The stability optimization technology of Change Data Capture (CDC) system is particularly suitable for enterprise-level database scenarios with high requirements for the reliability and real-time performance of data synchronization, including but not limited to CDC solutions for mainstream databases such as MySQL, PostgreSQL, and Oracle.

[0024] With the explosive growth of enterprise data and the widespread demand for real-time business operations, real-time data synchronization across systems has become the technological cornerstone for building data warehouses, implementing business disaster recovery, and conducting real-time data analysis. Against this backdrop, Change Data Capture (CDC) technology, with its advantages of low latency and low invasiveness, has become the mainstream solution for achieving real-time database synchronization.

[0025] Mainstream CDC (Data Change Detection) technologies are primarily based on database log parsing or data change event monitoring. Specifically, by parsing database transaction logs (such as MySQL's binary log (binlog) or PostgreSQL's write-ahead log (WAL), or capturing add, modify, and delete operation events in data tables, CDC technology can accurately capture incremental data and synchronize it in real time to various target systems (such as data lakes, data warehouses, or downstream business databases) without affecting the source database's business load. This technology is widely used in critical business scenarios such as data warehouse construction, heterogeneous database migration, cache updates, and ensuring eventual data consistency in microservice architectures.

[0026] Among related technologies, the CDC system suffers from the following technical problems, which seriously affect the system's stability and business continuity: (1) Lack of fault prediction capability and passive operation and maintenance response The existing system only focuses on data transmission status and does not perform multi-dimensional collaborative perception of database load (CPU / memory / number of connections), server resources (disk I / O ports / network bandwidth), and data acquisition link status (synchronization speed / latency / traffic). Alarms are only triggered after a failure occurs (such as source database crash or network interruption), making it impossible to predict risks in advance, resulting in increased business interruption time and high operation and maintenance costs.

[0027] (2) The consistency guarantee mechanism is inefficient and the testing cost is high. During system synchronization, inconsistencies can easily arise between the source and target databases due to network packet loss, task interruptions, and other anomalies. Traditional solutions rely on full data coverage or comparison for verification, which can take hours to days for large tables and consume significant computing and storage resources, making it difficult to meet the needs for regular and efficient integrity checks.

[0028] (3) The fault recovery methods are limited and the resynchronization efficiency is low. Fault recovery only supports two modes: "restart task" or "full resynchronization," and no tiered avoidance strategies (such as temporary speed reduction, task pause / resumption) are designed for different fault types. After a long pause, resynchronization requires repeated transmission of historical data, which not only results in low synchronization efficiency but also increases the load on the acquisition link, further prolonging the service recovery time.

[0029] To solve the above problems, such as Figure 1 As shown, this application provides a fault adaptive handling method for the entire process of a change data capture system, which includes at least steps S110 to S140: Step S110: Obtain multi-dimensional status data, which includes the running parameters and transmission parameters of each database instance in the change data capture system.

[0030] like Figure 2 As shown, the change data capture system includes a source database, a server, a target database, a network connecting the source database and the server, and a network connecting the server and the target database. Figure 2As shown, there can be multiple source databases, such as source database A, source database B, etc.; the server includes multiple service instances (data transmission software), such as service instance A, service instance B, etc.; and there can be multiple target databases, such as target database A, target database B, etc. During data transmission, a new piece of data is transmitted from source database A to the server via the network, and the server running service instance A transmits the new data to target database A. Similarly, a new piece of data is transmitted from source database B to the server via the network, and the server running service instance B transmits the new data to target database B. The operating parameters in the multiple source databases constitute the source data source indicators, the data generated by the multiple service instances running on the server constitutes the instance process indicators, the transmission indicators of new data between the source and target databases constitute the collected data indicators, and the operating parameters in the multiple target databases constitute the destination data source indicators.

[0031] The source and target databases are database instances in the change data capture system. Transmission parameters include the server's own operating parameters, instance process metrics generated by the server running different service instances, network transmission data of newly added data between the server and the source database, network transmission data of newly added data between the server and the target database, and data collection metrics during data transmission. The server is used for data collection or cluster synchronization. To achieve accurate fault prediction and diagnosis, real-time collection of multi-dimensional status data is required, covering the operating data of the source database, the operating data of the target database, server operating data (server's own parameters, instance process metrics), network transmission metrics, and data collection metrics. In the actual collection process, the specific items of the multi-dimensional status data are shown in Table 1. Table 1. Multi-dimensional status data of the change data capture system

[0032] Slow SQL refers to SQL statements that take longer than a preset threshold to execute. JVM memory refers to the memory area managed by the Java Virtual Machine during runtime. TCP is the Transmission Control Protocol, one of the most important and fundamental protocols in the TCP protocol suite, a core protocol of the Internet.

[0033] As shown in Table 1, the change data capture system in operation collects data every 5 seconds, which can obtain multi-dimensional status data from the source database and the target database, the server's own metrics, the data of the instance process during data transmission (such as the metrics of the programs running on the server), and the metrics generated by the collected data.

[0034] In one implementation, before inputting the multi-dimensional state data into a pre-trained time-to-all-domain-awareness fault detection model, the method further includes: cleaning the multi-dimensional state data to obtain intermediate data; and extracting and associating time-series features from the intermediate data to update the feature dimensions in the multi-dimensional state data.

[0035] The 3σ principle (also known as the Raida criterion) is a statistical method for identifying and eliminating outliers based on the distribution patterns of data.

[0036] Specifically, 3 The principle is to remove transient outliers from multi-dimensional status data, such as CPU suddenly jumping to 100% and then immediately recovering. For persistent outliers exceeding the corresponding threshold three times consecutively (e.g., server CPU ≥ 90%), these are marked as data to be monitored. Linear interpolation is used to supplement missing data within the collection period. If a data collector temporarily disconnects, causing a data entry to be missing, linear interpolation is used to fill in the missing value. If missing data occurs more than three times, a health check of the data collector is triggered, and the checked item is listed as data to be monitored. Complete multi-dimensional status data is converted into structured data, including the data collector's unique identifier, data type, data name, data value, unit, and collection time. Intermediate data is obtained through outlier removal, missing data supplementation, and data format conversion.

[0037] Using a fixed time window (e.g., 30 seconds, encompassing multiple data collection points), the mean, variance, trend slope (e.g., the rate of increase in CPU usage of the target database within 5 minutes), and peak percentage (e.g., the number of times the peak exceeds the 80% threshold within the window) of each indicator in the intermediate data are calculated to extract the temporal features of the intermediate data. By calculating the Pearson correlation coefficient (e.g., the correlation between server CPU utilization and acquisition task bandwidth) and temporal synchronization (e.g., the time difference between the increase in network latency and the increase in data synchronization latency) between two different types of data, feature vectors are constructed based on historical data to connect the temporal features of single indicators with the correlation features of multiple indicators, thereby optimizing and updating the feature dimensions of multi-dimensional state data.

[0038] S120. Input multi-dimensional state data into a pre-trained time-series-global perception fault detection model to identify and output fault information.

[0039] In one implementation, multi-dimensional state data is input into a pre-trained time-series-global awareness fault detection model to identify and output fault information. This includes: performing window sliding processing on the multi-dimensional state data based on the local feature extraction unit in the time-series-global awareness fault detection model to obtain short-term time-series data features; inputting the short-term time-series data features into the global awareness unit in the time-series-global awareness fault detection model to achieve global modeling through a cross-window attention mechanism; and performing multi-label classification processing on the global time-series data features to identify and output fault information.

[0040] Specifically, the time-series-global awareness fault detection model includes a local feature extraction unit (which can be a Long Short-Term Memory (LSTM) network) and a global awareness unit (which can be a deep learning model architecture with a self-attention mechanism). The local feature extraction unit inputs multi-dimensional state data into a short window, capturing short-term dynamics at the second to minute level to obtain short-term time-series data features. These features include instantaneous spikes in CPU usage, frequent garbage collection (GC), and sudden increases in network latency, enabling rapid alerts or rate limiting for anomalies. The short-term time-series data features are then input into the global awareness unit, which uses a medium-window input to model the cross-window global dependency of these features, thereby capturing long-term trends at the minute to hour level to obtain global time-series data features. Examples include continuously increasing system load and chronic expansion of database connections, enabling early warning. Multi-label classification processing is performed on the characteristics of time-series data across the entire domain to output fault information that represents anomalies in the data change capture system. The fault information includes fault type (database / server / instance process / network / acquisition link), real-time abnormal parameters, and expected fault occurrence time.

[0041] The parameters of the time-series-global awareness fault detection model during fault prediction are shown in Table 2: Table 2 Parameters of the Time-Series-Global Awareness Fault Detection Model

[0042] In one implementation, the construction steps of the time-series-global perception fault detection model include: acquiring training samples, which include running sample data, fault pre-sample data, and fault sample data; using the training samples to jointly train the local feature extraction unit and the global perception unit to obtain sample fault information; determining the loss value between the sample fault information and the actual fault information based on the loss function; and using the loss value to adjust the network parameters of the local feature extraction unit and the global perception unit until the loss value converges to a preset loss threshold to obtain the time-series-global perception fault detection model.

[0043] Specifically, historical operational data of the change data capture system is acquired as sample data. This sample data includes 70% operational sample data from when the change data capture system is running normally, used to learn system stability characteristics; 20% pre-fault sample data from before a failure occurs, containing indicator sequences from 1 to 3 minutes before the failure, showing the failure evolution trend; and 10% fault sample data from when the change data capture system fails, including labels such as fault type, fault cause, real-time abnormal parameters, and occurrence time, used to learn fault characteristics. Training samples are sequentially input into the local feature extraction unit and the global perception unit to extract short-term time-series data features from the training samples and output global time-series data features. The label type of the global time-series data features is analyzed to obtain sample fault information. The sample fault information and actual fault information are input into the loss function to obtain the loss value. Based on the loss value, the network parameters of the local feature extraction unit and the global perception unit are adjusted until the loss value converges and is less than or equal to a preset loss threshold, and the prediction accuracy is ≥95% and the misjudgment rate is ≤3%, thus obtaining the time-series-global perception fault detection model.

[0044] S130. Based on the fault type and real-time abnormal parameters in the fault information, execute the corresponding system recovery strategy.

[0045] In one implementation, a corresponding system recovery strategy is executed based on the fault type and real-time abnormal parameters in the fault information, including: determining the risk level according to the fault type and real-time abnormal parameters; matching a system recovery strategy from a preset strategy library according to the fault type, real-time abnormal parameters and risk level; wherein the system recovery strategy includes an associated feature chain that reflects the fault propagation path based on the fault information.

[0046] Specifically, the real-time anomaly parameters corresponding to each fault type are compared with the corresponding preset anomaly thresholds, and the risk level is determined by combining the anomaly duration; based on the fault type, real-time anomaly parameters, and risk level, the corresponding system recovery strategy is obtained from the preset strategy library.

[0047] Cross-dimensional indicator correlation verification of fault information improves predictive accuracy, avoids misjudgment based on a single indicator, and obtains a chain of related features. Example 1: "Server CPU ≥ 85%" + "Collection task bandwidth ≥ 60%" + "Other process indicators stable," determined as "High server CPU caused by collection task overload." Example 2: "Source database CPU ≥ 90%" + "Many slow SQL queries," determined as "High source database CPU caused by slow SQL queries." Example 3: "Source database CPU ≥ 90%" + "Other indicators stable," CPU recovers after pausing collection, then rises again after restarting, determined as "High source database CPU caused by collection tasks."

[0048] As shown in Table 3, system recovery strategies are implemented based on different fault types, fault causes and risk levels corresponding to real-time abnormal parameters. The warning information includes associated feature chains.

[0049] Table 3 System Recovery Strategy Execution Table

[0050] The execution of system recovery strategies can be adjusted manually or based on system recommendations.

[0051] S140. Evaluate the effectiveness of the system recovery strategy and dynamically optimize the parameters of the timing-global awareness fault detection model based on the effectiveness of the strategy, while adjusting the system recovery strategy.

[0052] In one implementation, evaluating the effectiveness of a system recovery strategy includes: monitoring real-time abnormal parameters corresponding to the system recovery strategy based on a preset sliding window; if the real-time abnormal parameters recover from an abnormal state to within the normal range or show a trend of recovery to the normal range, the effectiveness is determined to be valid; if the real-time abnormal parameters remain in an abnormal state or show a deteriorating trend, the effectiveness is determined to be invalid; if other related indicators, distinct from the real-time abnormal parameters, deteriorate, the effectiveness is determined to have side effects. Specifically, after the system recovery strategy is executed, continuous monitoring of real-time abnormal parameters is performed for 1 to 3 sampling periods. When the real-time abnormal parameters recover from an abnormal state to within the normal range or show a trend of recovery to the normal range, the effectiveness is determined to be valid. For example, if "the server's CPU is faulty, with a high risk level," and after implementing rate limiting for the data collection task, the CPU utilization drops below 80% or shows a trend of dropping below 80%, and the fluctuation of key indicators significantly decreases, the effectiveness is determined to be a valid action. If real-time abnormal parameters remain in an abnormal state or show a worsening trend, the execution effect is deemed ineffective. For example, if "the source database's CPU is at full capacity, with a high risk level," and after pausing the current database data collection task, CPU resources are not released immediately or CPU utilization approaches 100%, the source database's CPU anomaly is likely caused by other reasons, requiring intervention from operations personnel. Therefore, the execution effect is deemed ineffective. If, after executing the system recovery strategy, other related data (excluding real-time abnormal parameters) deteriorate, such as a continuous increase in data latency after rate limiting, the execution effect is deemed to have side effects. After completing the execution effect evaluation, the execution effect is recorded in the archived log, forming a structured data feedback set (prediction type, system recovery strategy, execution time, indicator changes, execution effect, etc.).

[0053] In one implementation, the parameters of the time-series-global awareness fault detection model are dynamically optimized based on the execution effect, and the system recovery strategy is adjusted, including: taking the execution effect determined to be effective as a positive sample and taking the execution effect determined to be invalid or have side effects as a negative sample; updating the training samples based on the positive and negative samples, and using the updated training samples to iteratively optimize the parameters of the time-series-global awareness fault detection model; and adjusting the execution action of the system recovery strategy corresponding to the negative sample based on the negative sample.

[0054] Specifically, effective execution results are used as positive samples, while ineffective or side-effect-laden execution results are used as negative samples. Training samples are updated at fixed intervals (e.g., daily / weekly) and incrementally fine-tuned. For categories with continuously increasing false positive rates (e.g., a false positive rate ≥5% for a certain type of fault), training samples are automatically added to supplement more contextual features from different time periods (e.g., process, network, or cache metrics). Based on the updated training samples, the parameters of the time-series-wide awareness fault detection model are iteratively optimized. After model optimization, the old and new models are run in parallel for a period. The prediction accuracy and the effectiveness of the system recovery strategy are compared through offline playback or online dual-parallel testing. The optimal time-series-wide awareness fault detection model is selected for further testing. The execution actions of the system recovery strategy are adjusted using negative samples to improve its effectiveness.

[0055] In one embodiment, the change data capture system includes a source database and a target database. The processing method further includes: obtaining the abnormal transmission time period corresponding to the fault information; performing multi-granularity consistency checks on the source database and the target database during the abnormal transmission time period; and performing data recovery operations in response to mismatched check results.

[0056] Specifically, the change data capture system includes a source database and a target database. The processing method further includes: when high-risk fault information exists, obtaining the start time of one or more recent abnormal archive logs related to the fault information; using the start time as the starting point and the current time as the ending point to determine the abnormal transmission period; performing consistency verification on the amount and content of newly added data in the source and target databases; and performing data recovery operations when the verification results are inconsistent. Alternatively, data verification can be performed periodically at fixed intervals. During periodic verification, a specific data transmission time segment is directly obtained to verify the consistency of the amount and content of newly added data in the source and target databases.

[0057] In one implementation, multi-granularity consistency checks are performed on the source and target databases, including: if the amount of new data added to the source database and the amount of new data added to the target database are the same, checking the summary data of the source and target databases to obtain a summary check result. Specifically, the amount of new data added to the source and target databases is compared; when the amount of new data added to the source and target databases is the same, the summary data of the source and target databases is checked to obtain a summary check result.

[0058] In one implementation, in response to a mismatched verification result, a data recovery operation is performed, including: performing data recovery on the target database based on the verification result of inconsistent new data volume; locating the discrepancy data based on the mismatched summary verification result, obtaining the archived log corresponding to the discrepancy data, merging the archived log and the discrepancy data, and writing the merged data into the target database; if the discrepancy data does not have an archived log, then synchronizing the discrepancy data from the source database to the target database.

[0059] Specifically, when the amount of newly added data in the source database and the target database is inconsistent, the newly added data sent from the source database is resent to supplement the data in the target database. When the digest verification result is mismatched, the difference data is located based on the digest verification result, the archived log of the difference data is obtained, the archived log is merged with the difference data, and the merged data is transmitted to the target database to supplement the data. If the archived log of the difference data has been cleaned up, the difference data is synchronized from the source database to the target database. For very large single tables, digest comparison avoids full scans and reduces the load on the source database.

[0060] This invention proposes an adaptive fault handling method for a change data capture system. The method includes: acquiring multi-dimensional state data of the change data capture system during operation; inputting the multi-dimensional state data into a pre-trained time-series-full-domain awareness fault detection model to identify and output fault information; executing corresponding system recovery strategies based on the fault type and real-time abnormal parameters in the fault information; evaluating the execution effect of the system recovery strategies, and dynamically optimizing the parameters of the time-series-full-domain awareness fault detection model based on the execution effect, while simultaneously adjusting the system recovery strategies. The processing method provided by this invention, by inputting real-time multi-dimensional state data into a pre-trained time-series-full-domain awareness fault detection model, achieves early prediction of system faults and executes different avoidance and recovery measures for different types of faults. It continuously optimizes the model and execution measures based on the execution effect of the recovery measures, achieving closed-loop monitoring. Furthermore, after a fault occurs, it quickly achieves consistent data comparison, resulting in high efficiency, low resource consumption, and periodic automated execution with high data integrity.

[0061] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A fault-based adaptive handling method for the entire process of a change data capture system, characterized in that, include: Acquire multi-dimensional status data, which includes the operating parameters and transmission parameters of each database instance in the change data capture system; The multi-dimensional state data is input into a pre-trained time-series-global awareness fault detection model to identify and output fault information; Based on the fault type and real-time abnormal parameters in the fault information, execute the corresponding system recovery strategy; The effectiveness of the system recovery strategy is evaluated, and the parameters of the time-series-global awareness fault detection model are dynamically optimized based on the effectiveness, while the system recovery strategy is adjusted.

2. The adaptive fault handling method for the change data capture system according to claim 1, characterized in that, Before inputting the multi-dimensional state data into the pre-trained time-series-global awareness fault detection model, the following steps are also included: The multi-dimensional state data is cleaned to obtain intermediate data; Temporal feature extraction and feature association are performed on the intermediate data to update the feature dimensions in the multi-dimensional state data.

3. The fault-based adaptive handling method for the entire process of a change data capture system according to claim 1, characterized in that, The multi-dimensional state data is input into a pre-trained time-series-global awareness fault detection model to identify and output fault information, including: Based on the local feature extraction unit in the time-series-global perception fault detection model, the multi-dimensional state data is subjected to window sliding processing to obtain short-term time-series data features. The short-term time-series data features are input into the global perception unit in the time-series-global perception fault detection model. Global modeling is achieved through a cross-window attention mechanism, and the global time-series data features are subjected to multi-label classification processing to identify and output the fault information.

4. The fault-based adaptive handling method for the entire process of a change data capture system according to claim 3, characterized in that, The construction steps of the time-series-global awareness fault detection model include: Acquire training samples, which include running sample data, pre-fault sample data, and fault sample data; The local feature extraction unit and the global perception unit are jointly trained using the training samples to obtain sample fault information. The loss value between the sample fault information and the actual fault information is determined based on the loss function. The network parameters of the local feature extraction unit and the global perception unit are adjusted using the loss value until the loss value converges to a preset loss threshold, thus obtaining the time-series-global perception fault detection model.

5. The fault-based adaptive handling method for the entire process of a change data capture system according to claim 4, characterized in that, Based on the fault type and real-time anomaly parameters in the fault information, execute the corresponding system recovery strategy, including: The risk level is determined based on the fault type and the real-time anomaly parameters. The system recovery strategy is matched from a preset strategy library based on the fault type, the real-time anomaly parameters, and the risk level. The system recovery strategy includes constructing an associated feature chain that reflects the fault propagation path based on the fault information.

6. The fault-based adaptive handling method for the entire process of a change data capture system according to claim 5, characterized in that, Evaluate the effectiveness of the system recovery strategy, including: Based on a preset sliding window, monitor real-time abnormal parameters corresponding to the system recovery strategy; If the real-time abnormal parameter recovers from the abnormal state to the normal range or if the real-time abnormal parameter shows a trend of recovering to the normal range, then the execution effect is determined to be effective. If the real-time abnormal parameter remains in the abnormal state or the real-time abnormal parameter shows a deteriorating trend, the execution effect is determined to be invalid. If other related indicators, distinct from the real-time anomaly parameters, deteriorate, then the execution effect is determined to have side effects.

7. The adaptive fault handling method for the change data capture system according to claim 6, characterized in that, Based on the execution results, the parameters of the time-series-global awareness fault detection model are dynamically optimized, and the system recovery strategy is adjusted, including: Execution results deemed valid are considered positive samples, while execution results deemed invalid or having side effects are considered negative samples. Based on the positive and negative samples, the training samples are updated, and the parameters of the time-series-global awareness fault detection model are iteratively optimized using the updated training samples. Based on the negative samples, adjust the execution actions of the system recovery strategy corresponding to the negative samples.

8. The fault-based adaptive handling method for the entire process of a change data capture system according to claim 1, characterized in that, The change data capture system includes a source database and a target database, and the processing method further includes: Obtain the abnormal transmission time period corresponding to the fault information; During the abnormal transmission period, multi-granularity consistency checks are performed on the source database and the target database, and data recovery operations are performed in response to mismatched check results.

9. The fault-based adaptive handling method for the entire process of a change data capture system according to claim 8, characterized in that, Perform multi-granularity consistency checks on the source database and the target database, including: If the amount of new data in the source database is the same as the amount of new data in the target database, the summary data of the source database and the target database are verified to obtain the summary verification result.

10. The fault-based adaptive handling method for the entire process of a change data capture system according to claim 9, characterized in that, In response to a mismatch in the verification result, perform data recovery operations, including: Data recovery is performed on the target database based on the verification results indicating inconsistent new data volume. Based on the mismatched summary verification results, locate the discrepancy data, obtain the archived log corresponding to the discrepancy data, merge the archived log with the discrepancy data, and write the merged data into the target database; If the difference data does not exist in the archived log, then the difference data is synchronized from the source database to the target database.

Images