A data full-link abnormality monitoring and processing method and system based on intelligent operation and maintenance

Abstract

This invention relates to the field of distributed data link monitoring technology, and discloses a method and system for monitoring and handling anomalies across the entire data link based on intelligent operation and maintenance. The method includes: acquiring multi-source operation and maintenance data and performing cross-link time normalization; identifying abnormal events based on a unified operation and maintenance feature sequence; constructing a delay causal graph carrying propagation delay based on the abnormal event sequence; performing counterfactual dynamic Bayesian inference based on the delay causal graph; and executing processing actions. Compared to existing technologies that rely on static thresholds for anomaly detection, especially under conditions of slow SQL, connection pool queuing, and mutual propagation of interface latency, this invention addresses the technical problem of failing to identify cross-component anomaly propagation relationships and achieve secure closed-loop processing. By identifying counterfactual root causes through anomaly event recognition and selecting security constraint processing actions, this application achieves accurate localization of the root causes of anomalies across the entire data link, improving the accuracy of anomaly response and operational stability of the distributed data link.

Description

Technical Field

[0001] This invention relates to the field of distributed data link monitoring technology, and in particular to a method and system for monitoring and handling anomalies in the entire data link based on intelligent operation and maintenance. Background Technology

[0002] Currently, in end-to-end data management scenarios such as databases, caches, and middleware, log monitoring and alerting rules based on fixed thresholds are commonly used. However, due to the varying sampling frequencies of operational data generated by different components and difficulties in timestamp alignment, alert information is often fragmented, making it difficult to form a unified time-based sequence. Furthermore, existing anomaly detection methods typically focus only on sudden increases in single-dimensional metrics, lacking consideration for multi-dimensional feature interactions and seasonal fluctuations, resulting in a high false alarm rate.

[0003] More importantly, when a cascading failure occurs in the system, such as cache penetration causing the database connection pool to become full, leading to a surge in interface latency and a simultaneous emergence of alarms at all levels, it becomes difficult for operations and maintenance personnel to quickly locate the root cause from a massive number of alarms. Traditional causal analysis methods cannot effectively handle indirect causal chains with latency, nor can they quantify the impact of different candidate root causes, resulting in low efficiency in root cause localization. Furthermore, the selection of automated processing actions lacks security constraints, which can easily introduce secondary risks. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, the purpose of this invention is to propose a data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance. This method aims to solve the technical problem that existing technologies rely on static thresholds for anomaly detection, especially under conditions of slow SQL, connection pool queuing, and interface latency propagation, which makes it impossible to identify cross-component anomaly propagation relationships and achieve secure closed-loop processing.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a method for monitoring and handling data anomalies across the entire data chain based on intelligent operation and maintenance.

[0006] The method for monitoring and handling data anomalies across the entire data chain based on intelligent operation and maintenance includes:

[0007] Step S10: Obtain multi-source operation and maintenance data, and perform cross-link time normalization task based on the multi-source operation and maintenance data using the exponential weighted moving average method to output a unified operation and maintenance feature sequence;

[0008] Step S20: Based on the unified operation and maintenance feature sequence, use a long short-term memory network and Holt Winters seasonal exponential smoothing method to perform anomaly event identification task and output anomaly event sequence;

[0009] Step S30: Based on the abnormal event sequence, the PCMCI causal discovery algorithm is used to perform a time-delay causal construction task, and a time-delay causal graph is output;

[0010] Step S40: Based on the time-delay causal graph, perform the root cause localization task using the counterfactual dynamic Bayesian inference method and output the root cause localization result;

[0011] Step S50: Select processing actions under the constraint of the safety barrier function based on the root cause localization results, and output the anomaly processing results.

[0012] Preferably, step S10, which involves acquiring multi-source operation and maintenance data, performing a cross-link time normalization task based on the multi-source operation and maintenance data using an exponentially weighted moving average method, and outputting a unified operation and maintenance feature sequence, specifically includes:

[0013] Step S101: Collect database audit logs, connection pool status data, cache slow logs and call chain data, and normalize the fields according to service identifier, instance identifier, database identifier, cache key prefix and call chain identifier to obtain normalized operation and maintenance records;

[0014] Step S102: Based on the normalized operation and maintenance records, extract the execution time, number of scanned rows, lock wait time, number of active connections, wait queue length, cache hit status, cache command time, and call span time to obtain the original operation and maintenance feature sequence;

[0015] Step S103: The original operation and maintenance feature sequence is smoothed and aligned under a unified sampling interval using the exponentially weighted moving average method, and the event occurrence time corresponding to each original operation and maintenance feature is retained to obtain the unified operation and maintenance feature sequence.

[0016] Preferably, step S20, which involves performing anomaly identification based on the unified operation and maintenance feature sequence using a long short-term memory network and HoltWinters seasonal exponential smoothing, and outputting the anomaly event sequence, specifically includes:

[0017] Step S201: Input the unified operation and maintenance feature sequence into the long short-term memory network to obtain a nonlinear prediction sequence;

[0018] Step S202: Input the unified operation and maintenance feature sequence into the Holt Winters seasonality index smoothing model to obtain the seasonal prediction sequence;

[0019] Step S203: Calculate the residual mutation intensity based on the nonlinear prediction sequence and the seasonal prediction sequence, and generate the abnormal event sequence according to the residual mutation intensity.

[0020] Preferably, step S203, which involves calculating the residual mutation intensity based on the nonlinear prediction sequence and the seasonal prediction sequence, and generating the anomalous event sequence based on the residual mutation intensity, includes:

[0021] The nonlinear prediction sequence and the seasonal prediction sequence are then fused using a load-aware weighted method to obtain the fused prediction value:

[0022]

[0023] in, For the first Maintenance and operation characteristics at any time The fusion prediction value, This represents the nonlinear predicted value output by the Long Short-Term Memory network. The seasonal forecast values ​​output by the Holt Winters seasonality index smoothing model. For load-aware fusion weights;

[0024] The load-aware fusion weights are determined as follows:

[0025]

[0026] in, For a moment The normalized value of the connection pool waiting queue. For a moment The normalized value of upstream request volume. As the first weighting coefficient, This is the second weighting coefficient. This is the bias coefficient;

[0027] Calculate the residual mutation intensity based on the fusion prediction value:

[0028]

[0029] in, For the first Maintenance and operation characteristics at any time The intensity of residual mutation, For the first Maintenance and operation characteristics at any time The measured value, The standard deviation of the residuals within the sliding window. It is a constant. For cache sensitivity, This represents the decrease in cache hit rate;

[0030] When the residual mutation intensity is greater than the quantile threshold of the corresponding dimension, the corresponding feature identifier, occurrence time, node to which it belongs, and residual mutation intensity are written into the abnormal event sequence.

[0031] Preferably, step S30, which involves using the PCMCI causal discovery algorithm based on the abnormal event sequence to perform a time-delay causal construction task and output a time-delay causal graph, specifically includes:

[0032] Step S301: Based on the node and occurrence time in the abnormal event sequence, construct a candidate causal node set, which includes execution plan nodes, lock waiting nodes, connection pool occupied nodes, cache hit nodes, upstream request volume nodes, and interface delay nodes;

[0033] Step S302: Use the PCMCI causal discovery algorithm to perform a lag conditional independence test on the candidate causal node set to obtain an initial set of directed causal edges;

[0034] Step S303: Based on the cache hit status, correct the causal propagation delay in the initial directed causal edge set to obtain the delay causal graph;

[0035] Among them, for the first The candidate causal node points to the first... The causal edges of the candidate causal nodes are calculated with the corrected propagation delay as follows:

[0036]

[0037] in, For a moment Corrected propagation delay, Based on historical propagation delay, Normalized value for call chain span time. This is the normalized value of the connection pool waiting queue. This is the normalized value for cache hit rate. For call chain latency weights, For connection pool queuing weights, This represents the weight of a cache miss.

[0038] Preferably, step S40, which involves performing the root cause localization task using the counterfactual dynamic Bayesian inference method based on the time-delay causal graph and outputting the root cause localization result, specifically includes:

[0039] Step S401: Write the abnormal event sequence as observation input into the time-delay causal graph, and construct a dynamic Bayesian network based on the time-delay causal graph to obtain a candidate root cause distribution containing candidate root cause nodes and their posterior probabilities;

[0040] Step S402: Perform counterfactual blocking assessment on the candidate root cause nodes in the candidate root cause distribution, calculate the abnormal decrease benefit after blocking the corresponding candidate root cause node, and obtain the counterfactual benefit value;

[0041] Step S403: Determine the target root cause node based on the candidate root cause nodes, and generate the root cause localization result based on the target root cause node;

[0042] The target root cause node is determined as follows:

[0043]

[0044] in, For the target root cause node, For the first One candidate root cause node For time-based At that time The posterior probabilities of candidate root causes are calculated from the observed anomaly sequences. For candidate root cause nodes Abnormally low returns after implementing counterfactual blocking.

[0045] Preferably, step S50, which involves selecting processing actions under the constraint of the safety barrier function based on the root cause localization result and outputting the anomaly processing result, specifically includes:

[0046] Step S501: Determine the object to be processed based on the root cause localization result, and construct an operation and maintenance status vector based on the object to be processed. The operation and maintenance status vector includes the root cause node identifier, root cause posterior probability, processor utilization rate, connection pool utilization rate, interface error rate, and transaction conflict risk value determined by the root cause localization result.

[0047] Step S502: Input the operation and maintenance state vector into the constrained Markov decision model to obtain the target processing action, which includes execution plan binding, connection pool parameter adjustment, cache preheating and statistical information refresh;

[0048] Step S503: Perform pre-execution verification on the target processing action based on the preset security barrier function to obtain the action security verification result, and execute the target processing action when the action security verification result satisfies the security barrier function constraint to obtain the anomaly processing result.

[0049] This invention also provides a data end-to-end anomaly monitoring and processing system based on intelligent operation and maintenance, comprising:

[0050] The time normalization module is used to acquire multi-source operation and maintenance data, and to perform cross-link time normalization tasks based on the multi-source operation and maintenance data using the exponential weighted moving average method, and output a unified operation and maintenance feature sequence.

[0051] The anomaly identification module is used to perform anomaly event identification tasks based on the unified operation and maintenance feature sequence using a long short-term memory network and HoltWinters seasonal exponential smoothing method, and output anomaly event sequence.

[0052] The time-delay causality construction module is used to perform a time-delay causality construction task based on the abnormal event sequence using the PCMCI causality discovery algorithm, and output a time-delay causality graph;

[0053] The root cause localization module is used to perform the root cause localization task based on the time-delay causal graph using the counterfactual dynamic Bayesian inference method, and output the root cause localization result.

[0054] The exception handling module is used to select processing actions under the constraints of the safety barrier function based on the root cause localization results, and output the exception handling results.

[0055] The present invention also provides a data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance. The data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance includes: a memory, a processor, and a data end-to-end anomaly monitoring and processing program based on intelligent operation and maintenance stored in the memory and executable on the processor. When the data end-to-end anomaly monitoring and processing program based on intelligent operation and maintenance is executed by the processor, it implements the above-mentioned method.

[0056] The present invention also provides a computer program product, the computer program product including a data end-to-end anomaly monitoring and processing program based on intelligent operation and maintenance, the data end-to-end anomaly monitoring and processing program based on intelligent operation and maintenance implementing the above method when executed by a processor.

[0057] The beneficial effects of this invention are as follows:

[0058] 1. This invention performs cross-link time normalization on multi-source operation and maintenance data using the exponential weighted moving average method, which solves the problem of timestamp misalignment caused by inconsistent sampling frequencies, provides a unified time reference sequence for subsequent anomaly detection, and effectively avoids time sequence misalignment during cross-source data fusion.

[0059] 2. This invention uses a long short-term memory network and Holt Winters seasonal exponential smoothing method for anomaly event identification. It can capture complex nonlinear patterns and extract periodic seasonal patterns. By calculating the residual mutation intensity through load-aware weighted fusion, it significantly improves the accuracy and robustness of anomaly event identification, especially when business traffic fluctuates and cache status changes.

[0060] 3. This invention utilizes the PCMCI causal discovery algorithm to construct a latency causal graph, and dynamically corrects the propagation latency based on call chain time, connection pool queuing, and cache hit status, making the fault propagation chain more closely resemble the real production environment and providing a reliable causal structure for root cause localization. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the first embodiment of a data end-to-end anomaly monitoring and handling method based on intelligent operation and maintenance according to the present invention. Detailed Implementation

[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0063] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of a data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance according to the present invention.

[0065] In the first embodiment, the data end-to-end anomaly monitoring and handling method based on intelligent operation and maintenance includes:

[0066] Step S10: Obtain multi-source operation and maintenance data, and perform cross-link time normalization task based on the multi-source operation and maintenance data using the exponential weighted moving average method to output a unified operation and maintenance feature sequence;

[0067] The multi-source operation and maintenance data in this step specifically includes database audit logs, connection pool status data, cache slow logs, and call chain data. After obtaining normalized operation and maintenance records through field normalization, execution time, number of rows scanned, lock wait time, number of active connections, wait queue length, cache hit status, cache command time, and call span time are extracted to form the original operation and maintenance feature sequence. The exponentially weighted moving average method is used to smoothly align each original feature sequence based on a uniform sampling interval, while retaining the event occurrence time. The output unified operation and maintenance feature sequence serves as the input for subsequent abnormal event identification.

[0068] EWMA is used to normalize the time of multi-source heterogeneous data, eliminating the time base deviation caused by the difference in sampling periods of different components. This aligns discrete and asynchronous operation and maintenance features to the same time grid, making subsequent cross-link feature correlation analysis possible, thus laying a unified time dimension foundation for end-to-end anomaly detection.

[0069] Traditional methods often process each data source individually or use simple interpolation for alignment, which is insufficient to handle scenarios with significantly different sampling frequencies, frequently leading to missed or false alarms due to time misalignment. This step uses an exponentially weighted moving average for smooth alignment, which preserves trend information while suppressing noise. Compared to fixed-interval resampling, it better reflects instantaneous characteristics, providing more accurate sequence input for subsequent models.

[0070] For example, the database audit log generates one record every second, while the cache slow log is output every 5 seconds. By using an exponentially weighted moving average with a uniform interval of 1 second, the 5-second sampling points of the cache slow log are exponentially decayed and smoothed to obtain an approximate cache command consumption sequence per second. At the same time, multiple database features are also smoothed to the same moment. In subsequent anomaly identification, the correlation between the number of database scan rows and the cache response time can be compared at the same point in time, and short-term sudden anomalies will not be missed due to different sampling step sizes.

[0071] Step S20: Based on the unified operation and maintenance feature sequence, use a long short-term memory network and Holt Winters seasonal exponential smoothing method to perform anomaly event identification task and output anomaly event sequence;

[0072] This step inputs the unified operation and maintenance feature sequence into a Long Short-Term Memory (LSTM) network and a Holt Winters seasonality exponential smoothing model, respectively, to obtain nonlinear and seasonal prediction sequences. Then, load-aware weighted fusion is used to generate a fused prediction value, and the residual mutation intensity is calculated. The fusion weights are dynamically adjusted based on factors such as the connection pool waiting queue and upstream request volume. When the residual mutation intensity exceeds the quantile threshold of the corresponding dimension, the feature identifier, occurrence time, associated node, and residual mutation intensity are written into the abnormal event sequence.

[0073] By fusing predictions using dual models, the system combines the powerful fitting ability of LSTM for nonlinear dependencies with the effective decomposition of seasonality and trends by HoltWinters. The load-aware weights enable the model to adaptively adjust the prediction focus under different load conditions, such as peak traffic periods and decreased cache hit rates, thereby improving the sensitivity and specificity of anomaly detection.

[0074] Compared to single models or static thresholds, this method does not generate a large number of false alarms when business traffic fluctuates or cache expires, and can also capture slowly evolving seasonal anomalies. In particular, by introducing cache sensitivity coefficients and connection pool queuing information to correct residual mutation intensity, slow faults related to cache penetration and connection pool saturation are more easily detected, making up for the shortcomings of traditional methods that rely solely on absolute bias.

[0075] For example, if the upstream request volume surges during a certain period, the connection pool waiting queue value increases, and the load-aware fusion weight will tilt towards LSTM because non-linear mutations dominate at this time. At the same time, if the cache hit rate suddenly drops significantly, the cache sensitivity coefficient amplifies the residual mutation intensity. Even if the absolute value of the time increase is not large, it can trigger the writing of abnormal events, thereby timely capturing the response degradation caused by the expiration of cache hot keys.

[0076] Step S30: Based on the abnormal event sequence, the PCMCI causal discovery algorithm is used to perform a time-delay causal construction task, and a time-delay causal graph is output;

[0077] This step constructs a candidate causal node set based on the node and occurrence time in the sequence of abnormal events, including execution plan nodes, lock waiting nodes, connection pool occupied nodes, cache hit nodes, upstream request volume nodes, and interface delay nodes. The PCMCI causal discovery algorithm is used to perform lag conditional independence checks on the candidate nodes, obtaining initial directed causal edges; then, cache hit status is used to correct the causal propagation delay, generating a delay causal graph.

[0078] The time-delay causal graph constructed in this way can not only reveal the direction of fault propagation, but also depict the lag effect caused by resource contention, network latency and other factors in the actual system by dynamically correcting the propagation delay. This more realistically reflects the temporal relationship of the fault chain and provides a refined causal structure for root cause localization.

[0079] Traditional causal discovery methods often assume fixed latency or ignore latency variations, making them ill-suited to the dynamic latency caused by elastic scaling and sudden traffic surges in cloud computing environments. This invention combines call chain latency, connection pool queuing, and cache hit rate to correct causal edge latency in real time, enabling the causal graph to adaptively adjust as the system state evolves, significantly improving the accuracy of causal discovery.

[0080] For example, suppose a sudden change in the database execution plan causes a surge in the number of rows scanned, leading to increased lock waiting and connection pool usage, ultimately increasing interface latency. The PCMCI algorithm identifies the causal chain from the execution plan node to the lock waiting node, then to the connection pool node, and finally to the interface latency node. During correction, if the call chain span is very long and the connection pool queue is high, the algorithm can accurately pinpoint the lag time of fault propagation on the latency causal graph, facilitating accurate assessment of timing relationships in the next step of root cause localization.

[0081] Step S40: Based on the time-delay causal graph, perform the root cause localization task using the counterfactual dynamic Bayesian inference method and output the root cause localization result;

[0082] This step involves writing the sequence of anomalous events into a time-delayed causal graph to construct a dynamic Bayesian network, obtaining candidate root cause nodes and their posterior probability distributions; performing counterfactual blocking evaluation on each candidate root cause and calculating the anomalous decrease gain after blocking; then determining the target root cause node based on the product of the posterior probability and the counterfactual gain value, and generating the root cause localization result.

[0083] By combining observational evidence with causal structure for dynamic Bayesian inference, the quantitative confidence of each candidate root cause can be obtained. At the same time, the counterfactual payoff value assesses the degree of mitigation of system chain anomalies after isolating a certain node. The joint decision-making of the two makes the localization result both consistent with statistical evidence and have the greatest repair value.

[0084] Existing root cause localization methods often rely on correlation analysis or probability ranking, failing to consider the global benefits of actually blocking a particular node. This may lead to minor, superficial anomalies being identified as root causes. This invention introduces counterfactual reasoning to evaluate the benefits of anomaly reduction, distinguishing between source faults and propagated faults, avoiding ineffective corrective actions, and improving the accuracy and practicality of root cause localization.

[0085] For example, in a scenario where cache misses cause a surge in database pressure, the dynamic Bayesian network gives a posterior probability of 0.7 for cache-hit nodes and 0.8 for connection pool-occupied nodes. However, counterfactual analysis shows that blocking cache-hit nodes can reduce the interface error rate by approximately 60%, a significant improvement; while blocking connection pool nodes only reduces it by 10%. The ultimate root cause (R*) will select cache-hit nodes because of their higher overall score, thus guiding operations to prioritize fixing cache issues and quickly restore service.

[0086] Step S50: Select processing actions under the constraint of the safety barrier function based on the root cause localization results, and output the anomaly processing results.

[0087] This step determines the objects to be processed based on the root cause localization results, and constructs an operation and maintenance state vector containing root cause node identifiers, posterior probabilities, processor utilization, connection pool utilization, interface error rates, and transaction conflict risk values. This vector is then input into a constrained Markov decision model, which outputs target processing actions such as execution plan binding, connection pool parameter adjustment, cache preheating, and statistics refresh. The actions are then verified for security by a preset security barrier function. Once the verification is successful, the actions are executed and the anomaly handling results are obtained.

[0088] By inputting the operation and maintenance state vector into the Markov decision model, the current system load and risk can be comprehensively considered to select the optimal repair strategy; the safety barrier function performs the final layer of risk isolation before execution to prevent the repair action from causing additional failures in the critical state, and ensures the reliability and safety of automatic processing.

[0089] Existing automatic repair methods often directly call predefined scripts after an anomaly is triggered, lacking the ability to assess the system's current capacity and intercept high-risk actions. This may lead to cache warm-up or connection pool adjustments during peak business hours, causing temporary response degradation or even service interruption. This invention uses a security barrier function to dynamically constrain actions. When a transaction conflict risk or interface error rate exceeds a threshold, execution is suspended or downgraded, reducing the risk of secondary incidents.

[0090] For example, when the root cause is identified as an abnormal database execution plan, the model selects the "execution plan binding" action. The security barrier function checks that the current transaction conflict risk value is as high as 0.9 (close to the deadlock threshold). At this point, the action is determined to be high-risk, so the binding is temporarily suspended. Instead, the low-risk action of "statistics refresh" is triggered, and the high-risk action is added to the delay queue to wait for the load to decrease before execution. This avoids large lock waits caused by execution plan switching in transaction-intensive scenarios.

[0091] Example 2: Furthermore, the present invention provides a data end-to-end anomaly monitoring and processing system based on intelligent operation and maintenance, employing a data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in the above embodiments, which can solve the technical problem of data end-to-end anomaly monitoring and processing based on intelligent operation and maintenance. The beneficial effects of the data end-to-end anomaly monitoring and processing system based on intelligent operation and maintenance provided by the present invention are the same as those of the data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance provided in the above embodiments, and other technical features of the data end-to-end anomaly monitoring and processing system based on intelligent operation and maintenance are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0092] Example 3: This invention provides a data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which are then executed to enable the at least one processor to perform the data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance described in Example 1. The data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance is merely an example and should not impose any limitations on the functionality or scope of use of this invention. A data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory or a program loaded from a storage device into random access memory. The random access memory also stores various programs and data required for the operation of the data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance. The processing unit, read-only memory, and random access memory are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance with various systems, it should be understood that it is not required to implement or possess all the systems shown. It can be implemented alternatively or with more or fewer systems.

[0093] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for monitoring and handling data anomalies across the entire data link based on intelligent operation and maintenance. The computer program product provided by this invention can solve the technical problem of monitoring and handling data anomalies across the entire data link based on intelligent operation and maintenance. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the method for monitoring and handling data anomalies across the entire data link based on intelligent operation and maintenance provided in the above embodiments, and will not be repeated here.

[0094] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0095] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0096] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for monitoring and handling data anomalies across the entire data chain based on intelligent operation and maintenance, characterized in that, The methods include: Step S10: Obtain multi-source operation and maintenance data, and perform cross-link time normalization task based on the multi-source operation and maintenance data using the exponential weighted moving average method to output a unified operation and maintenance feature sequence; Step S20: Based on the unified operation and maintenance feature sequence, use a long short-term memory network and Holt Winters seasonal exponential smoothing method to perform anomaly event identification task and output anomaly event sequence; Step S30: Based on the abnormal event sequence, the PCMCI causal discovery algorithm is used to perform a time-delay causal construction task, and a time-delay causal graph is output; Step S40: Based on the time-delay causal graph, perform the root cause localization task using the counterfactual dynamic Bayesian inference method and output the root cause localization result; Step S50: Select processing actions under the constraint of the safety barrier function based on the root cause localization results, and output the anomaly processing results.

2. The data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in claim 1, characterized in that, Step S10 involves acquiring multi-source operation and maintenance data, performing a cross-link time normalization task based on the multi-source operation and maintenance data using the exponentially weighted moving average method, and outputting a unified operation and maintenance feature sequence. Specifically, this includes: Step S101: Collect database audit logs, connection pool status data, cache slow logs and call chain data, and normalize the fields according to service identifier, instance identifier, database identifier, cache key prefix and call chain identifier to obtain normalized operation and maintenance records; Step S102: Based on the normalized operation and maintenance records, extract the execution time, number of scanned rows, lock wait time, number of active connections, wait queue length, cache hit status, cache command time, and call span time to obtain the original operation and maintenance feature sequence; Step S103: The original operation and maintenance feature sequence is smoothed and aligned under a unified sampling interval using the exponentially weighted moving average method, and the event occurrence time corresponding to each original operation and maintenance feature is retained to obtain the unified operation and maintenance feature sequence.

3. The data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in claim 1, characterized in that, Step S20, which involves using a Long Short-Term Memory network and Holt-Winters seasonal exponential smoothing method to perform anomaly identification based on the unified operation and maintenance feature sequence and outputting the anomaly event sequence, specifically includes: Step S201: Input the unified operation and maintenance feature sequence into the long short-term memory network to obtain a nonlinear prediction sequence; Step S202: Input the unified operation and maintenance feature sequence into the Holt Winters seasonality index smoothing model to obtain the seasonal prediction sequence; Step S203: Calculate the residual mutation intensity based on the nonlinear prediction sequence and the seasonal prediction sequence, and generate the abnormal event sequence according to the residual mutation intensity.

4. The data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in claim 3, characterized in that, Step S203, which involves calculating the residual mutation intensity based on the nonlinear prediction sequence and the seasonal prediction sequence, and generating the anomalous event sequence based on the residual mutation intensity, includes: The nonlinear prediction sequence and the seasonal prediction sequence are then fused using a load-aware weighted method to obtain the fused prediction value: in, For the first Maintenance and operation characteristics at any time The fusion prediction value, This represents the nonlinear predicted value output by the Long Short-Term Memory network. The seasonal forecast values ​​output by the Holt Winters seasonality index smoothing model. For load-aware fusion weights; The load-aware fusion weights are determined as follows: in, For a moment The normalized value of the connection pool waiting queue. For a moment The normalized value of upstream request volume. As the first weighting coefficient, This is the second weighting coefficient. This is the bias coefficient; Calculate the residual mutation intensity based on the fusion prediction value: in, For the first Maintenance and operation characteristics at any time The intensity of residual mutation, For the first Maintenance and operation characteristics at any time The measured value, The standard deviation of the residuals within the sliding window. It is a constant. For cache sensitivity, This represents the decrease in cache hit rate; When the residual mutation intensity is greater than the quantile threshold of the corresponding dimension, the corresponding feature identifier, occurrence time, node to which it belongs, and residual mutation intensity are written into the abnormal event sequence.

5. The data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in claim 1, characterized in that, Step S30, which involves using the PCMCI causal discovery algorithm based on the abnormal event sequence to perform a time-delay causal construction task and output a time-delay causal graph, specifically includes: Step S301: Based on the node and occurrence time in the abnormal event sequence, construct a candidate causal node set, which includes execution plan nodes, lock waiting nodes, connection pool occupied nodes, cache hit nodes, upstream request volume nodes, and interface delay nodes; Step S302: Use the PCMCI causal discovery algorithm to perform a lag conditional independence test on the candidate causal node set to obtain an initial set of directed causal edges; Step S303: Based on the cache hit status, correct the causal propagation delay in the initial directed causal edge set to obtain the delay causal graph; Among them, for the first The candidate causal node points to the first... The causal edges of the candidate causal nodes are calculated with the corrected propagation delay as follows: in, For a moment Corrected propagation delay, Based on historical propagation delay, Normalized value for call chain span time. This is the normalized value of the connection pool waiting queue. This is the normalized value for cache hit rate. For call chain latency weights, For connection pool queuing weights, This represents the weight of a cache miss.

6. The data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in claim 1, characterized in that, Step S40, which involves performing the root cause localization task using the counterfactual dynamic Bayesian inference method based on the time-delay causal graph and outputting the root cause localization result, specifically includes: Step S401: Write the abnormal event sequence as observation input into the time-delay causal graph, and construct a dynamic Bayesian network based on the time-delay causal graph to obtain a candidate root cause distribution containing candidate root cause nodes and their posterior probabilities; Step S402: Perform counterfactual blocking assessment on the candidate root cause nodes in the candidate root cause distribution, calculate the abnormal decrease benefit after blocking the corresponding candidate root cause node, and obtain the counterfactual benefit value; Step S403: Determine the target root cause node based on the candidate root cause nodes, and generate the root cause localization result based on the target root cause node; The target root cause node is determined as follows: in, For the target root cause node, For the first One candidate root cause node For time-based At that time The posterior probabilities of candidate root causes are calculated from the observed anomaly sequences. For candidate root cause nodes Abnormally low returns after implementing counterfactual blocking.

7. The data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in claim 1, characterized in that, Step S50, which involves selecting processing actions under the constraint of the safety barrier function based on the root cause localization result and outputting the anomaly handling result, specifically includes: Step S501: Determine the object to be processed based on the root cause localization result, and construct an operation and maintenance status vector based on the object to be processed. The operation and maintenance status vector includes the root cause node identifier, root cause posterior probability, processor utilization rate, connection pool utilization rate, interface error rate, and transaction conflict risk value determined by the root cause localization result. Step S502: Input the operation and maintenance state vector into the constrained Markov decision model to obtain the target processing action, which includes execution plan binding, connection pool parameter adjustment, cache preheating and statistical information refresh; Step S503: Perform pre-execution verification on the target processing action based on the preset security barrier function to obtain the action security verification result, and execute the target processing action when the action security verification result satisfies the security barrier function constraint to obtain the anomaly processing result.

8. A data end-to-end anomaly monitoring and processing system based on intelligent operation and maintenance, applied to the data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in any one of claims 1 to 7, characterized in that, The data end-to-end anomaly monitoring and processing system includes: The time normalization module is used to acquire multi-source operation and maintenance data, and to perform cross-link time normalization tasks based on the multi-source operation and maintenance data using the exponential weighted moving average method, and output a unified operation and maintenance feature sequence. The anomaly identification module is used to perform anomaly event identification tasks based on the unified operation and maintenance feature sequence using a long short-term memory network and Holt Winters seasonal exponential smoothing method, and output anomaly event sequence. The time-delay causality construction module is used to perform a time-delay causality construction task based on the abnormal event sequence using the PCMCI causality discovery algorithm, and output a time-delay causality graph; The root cause localization module is used to perform the root cause localization task based on the time-delay causal graph using the counterfactual dynamic Bayesian inference method, and output the root cause localization result. The exception handling module is used to select processing actions under the constraints of the safety barrier function based on the root cause localization results, and output the exception handling results.

9. A data end-to-end anomaly monitoring and processing device based on intelligent operation and maintenance, characterized in that, The intelligent operation and maintenance-based data full-link anomaly monitoring and processing device includes: a memory, a processor, and an intelligent operation and maintenance-based data full-link anomaly monitoring and processing program stored in the memory and executable on the processor. When the intelligent operation and maintenance-based data full-link anomaly monitoring and processing program is executed by the processor, it implements the intelligent operation and maintenance-based data full-link anomaly monitoring and processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a data end-to-end anomaly monitoring and processing program based on intelligent operation and maintenance. When the data end-to-end anomaly monitoring and processing program based on intelligent operation and maintenance is executed by the processor, it implements a data end-to-end anomaly monitoring and processing method based on intelligent operation and maintenance as described in any one of claims 1 to 7.

Images