Alarm root cause positioning method, system and equipment of arrangement system

Abstract

The invention discloses an alarm root cause positioning method, system and device for an arrangement system, and relates to the technical field of computers, and the method comprises the steps: collecting a link index data set and a component index data set in the arrangement system; on the basis of a current threshold self-adaptive mechanism, performing anomaly identification according to the link index data set and the component index data set, and determining an index anomaly event set; based on a preset link-component mapping table, in combination with the index exception event set, determining a link and component association exception event; and performing fusion processing, alarm processing and root cause identification on the multi-source abnormal events according to the link and component association abnormal events based on the component topology dependency relationship, and determining an aggregation alarm event and a root cause positioning report. According to the invention, accurate early warning and rapid root cause positioning can be realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, system and device for locating the root cause of alarms in an orchestration system. Background Technology

[0002] With the widespread adoption of distributed system architectures, orchestration systems, as the core of business processes and resource scheduling, directly determine the availability of upper-layer services based on their stability. System monitoring and alerting have become core aspects of distributed architecture operations and maintenance. Current technological developments exhibit two main trends: first, an extension from single-point monitoring to link monitoring, focusing on anomaly detection throughout the entire message flow path; and second, an upgrade from resource monitoring to layered component monitoring, covering full-stack metrics across the business layer, core capability layer, and basic support layer. However, existing technologies have significant shortcomings: (1) Single message link monitoring technology: Lack of linkage mechanism with component monitoring: This technology only focuses on link metrics. When message backlog occurs in the link, it cannot directly correlate with the resource status of the corresponding processing component (such as JVM memory overflow, high CPU utilization). Operations personnel must manually troubleshoot across systems, resulting in low localization efficiency. Poor adaptability of static thresholds: Thresholds are fixed values ​​and cannot be dynamically adjusted according to load changes during peak / off-peak periods, easily leading to "false alarms during off-peak periods and missed alarms during peak periods." No alarm aggregation capability: A failure at one node in the link can trigger backlog alarms from multiple downstream nodes, creating an alarm storm that interferes with the judgment of operations personnel.

[0003] (2) Layered component monitoring technology: Lack of end-to-end link perspective: This technology only focuses on the state of the component itself and cannot correlate the impact of component anomalies on the message chain (e.g., performance degradation of the basic assurance layer cache will lead to a decrease in the message consumption rate of core capability layer business components), making full-process traceability impossible. No dynamic fluctuation detection capability: It only supports anomaly judgment based on fixed thresholds and cannot identify trend anomalies in metrics. Weak root cause localization capability: It can only mark the hierarchical impact range of the abnormal component and cannot determine the source of the fault through time-series correlation analysis.

[0004] These shortcomings make it difficult for existing monitoring solutions to meet the operational and maintenance requirements of "accurate early warning and rapid location" in complex orchestration systems with high concurrency and strong dependencies. Summary of the Invention

[0005] The purpose of this application is to provide an alarm root cause localization method, system and device for an orchestration system, which can achieve accurate early warning and rapid root cause localization.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for locating the root cause of alarms in an orchestration system, including: Collect link metric datasets and component metric datasets from the orchestration system; Load the current threshold adaptive mechanism, and based on the current threshold adaptive mechanism, perform anomaly identification according to the link indicator dataset and the component indicator dataset to determine the set of indicator anomaly events; Based on the preset link-component mapping table and combined with the set of abnormal events of the indicators, abnormal events associated with links and components are determined; Based on the component topology dependency, and according to the abnormal events associated with the links and components, multi-source abnormal events are fused, alarms are processed, and root causes are identified to determine aggregated alarm events and root cause location reports.

[0007] Secondly, this application provides an alarm root cause localization system for an orchestration system, comprising: The dual-dimensional monitoring data acquisition module is used to collect link indicator datasets and component indicator datasets in the orchestration system; An anomaly identification module is used to load the current threshold adaptive mechanism and, based on the current threshold adaptive mechanism, perform anomaly identification according to the link indicator dataset and the component indicator dataset to determine the set of indicator anomaly events. The dual-dimensional linkage analysis module is used to determine the abnormal events associated with links and components based on a preset link-component mapping table and the set of abnormal events of the indicators. The alarm aggregation root cause analysis module is used to perform multi-source abnormal event fusion processing, alarm processing, and root cause identification based on the component topology dependency relationship and the abnormal events associated with the link and component, and to determine the aggregated alarm events and root cause location report.

[0008] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an alarm root cause localization method for an orchestration system.

[0009] According to the specific embodiments provided in this application, the following technical effects are disclosed: Based on the link indicator dataset and component indicator dataset in the orchestration system, this application participates in subsequent anomaly identification, realizing bidirectional linkage analysis of "link anomaly - component location" and "component anomaly - link impact," thus solving the problem of independent link monitoring and component monitoring. This application sets a current threshold adaptive mechanism for anomaly identification, solving the problem of poor adaptability of static thresholds. Dynamic fluctuation detection can be implemented to achieve accurate identification of indicator anomalies. This application sets a preset link-component mapping table to achieve data interaction in two dimensions, identifying link-component associated anomaly events, thereby achieving full-link and full-level monitoring coverage of the message chain and components, solving the problem of insufficient end-to-end monitoring coverage. Finally, based on component topology dependencies and link-component associated anomaly events, multi-source anomaly event fusion processing, alarm processing, and root cause identification are performed, solving the alarm storm problem. Through alarm aggregation and suppression strategies, the root cause of the fault is automatically, quickly, and accurately identified. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a diagram illustrating the application environment of the alarm root cause localization method of the orchestration system in one embodiment of this application.

[0012] Figure 2 This is a flowchart illustrating the alarm root cause localization method of the orchestration system in one embodiment of this application.

[0013] Figure 3 This is a schematic diagram of the alarm root cause localization system of the orchestration system in one embodiment of this application.

[0014] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] This application adopts a two-dimensional architecture of "message link monitoring dimension + component module hierarchical monitoring dimension". Through linkage analysis, it realizes data interaction between the two dimensions, and combines dynamic baseline calculation and alarm aggregation root cause analysis to complete accurate early warning and root cause location.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] The alarm root cause localization method for the orchestration system provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on another server. Terminal 101 can send the link indicator dataset and component indicator dataset from the orchestration system to server 102. After receiving them, server 102 performs anomaly identification based on the current threshold adaptive mechanism to determine the set of indicator anomaly events; it combines a preset link-component mapping table with the set of indicator anomaly events to determine link-component related anomaly events; based on component topology dependencies and link-component related anomaly events, it performs multi-source anomaly event fusion processing, alarm processing, and root cause identification to determine aggregated alarm events and root cause location reports. Server 102 can feed back the obtained aggregated alarm events and root cause location reports to terminal 101. Furthermore, in some embodiments, the alarm root cause location method of the orchestration system can also be implemented independently by server 102 or terminal 101.

[0019] The terminal 101 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 102 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0020] In one exemplary embodiment, such as Figure 2 As shown, a method for locating the root cause of alarms in an orchestration system is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps 201 to 204.

[0021] Step 201: Collect the link indicator dataset and component indicator dataset from the orchestration system.

[0022] The orchestration system involves message chain links, message relay caches, communication services, message brokers, and business processing components. The message chain link refers to the complete flow path of messages from the client to the backend components, including key nodes such as the client, communication service, message broker, message relay cache, and business processing components. The message relay cache is a cache middleware in the message chain used to temporarily store messages to be processed. The communication service refers to the service component responsible for communication between the client and the backend system. The message broker is a middleware component responsible for message routing and distribution. The business processing components are the components that actually process business logic, such as core business processing components and multi-source data processing components.

[0023] In one specific application, step 201 includes: (11) Collect the original link indicator data of each link node in the orchestration system; the link node is one of the client, communication service, message broker, message relay cache and business processing components, and the original link indicator data includes four core indicators: queue backlog, message processing rate, message flow success rate and message delay.

[0024] (12) Collect raw component indicator data of each component layer in the orchestration system; the component layer is one of the business association layer (management backend / frontend portal), core capability layer (core business processing component / multi-source data processing component) and basic guarantee layer (system resources / database / JVM); the raw component indicator data covers resource indicators, business indicators and health status indicators.

[0025] (13) Standardize all the original link indicator data and the original component indicator data to obtain the link indicator dataset and the component indicator dataset. The standardization process includes unifying the time granularity to 10s and filtering outliers.

[0026] If step 201 is considered as a two-dimensional monitoring data acquisition module, the input is the raw operating data of each node / component, and the output is a standardized link indicator dataset and component indicator dataset. The two-dimensional monitoring data acquisition module includes a message link acquisition unit, a component layered acquisition unit, and an indicator preprocessing unit. The message link acquisition unit executes step (11) and is deployed on link nodes such as clients, communication services, message brokers, message relay caches, and business processing components; the component layered acquisition unit executes step (12) and is deployed on the business association layer, core capability layer, and basic guarantee layer; the indicator preprocessing unit executes step (13).

[0027] Step 202: Load the current threshold adaptive mechanism, and based on the current threshold adaptive mechanism, perform anomaly identification according to the link indicator dataset and the component indicator dataset to determine the set of indicator anomaly events.

[0028] In one specific application, step 202 includes: (21) For any metric in the link metric dataset and the component metric dataset, determine the current threshold, current window, and current deviation judgment rule based on the current threshold adaptive mechanism. Specifically, the baseline calculation window and deviation threshold are automatically adjusted according to the business load characteristics, and different deviation judgment rules can be configured for different metric types.

[0029] In a specific application, the following steps are included: 1) Call the corresponding deviation judgment rule according to the type of the indicator, and use it as the current deviation judgment rule.

[0030] This application provides a differentiated adjustment mechanism based on indicator type: different types of indicators are classified according to their characteristics. For example, backlog indicators (such as queue length) are calculated using a deviation rate threshold; rate indicators (such as transactions per second (TPS) and processing rate) are judged based on abnormal growth rate; and availability indicators (such as health score and success rate) are detected using a fixed lower limit or a continuous downward trend.

[0031] 2) Retrieve the time series of the indicator and perform moving average analysis and cycle identification to determine the volatility. This is the window adaptive mechanism based on load cycle patterns provided in this application: perform moving average analysis and cycle identification algorithms (e.g., based on FFT or autocorrelation function) on the indicator time series to identify business peak and trough patterns and calculate volatility.

[0032] 3) When the volatility is in the first fluctuation range, increase the preset window to obtain the current window; when the volatility is in the second fluctuation range, decrease the preset window to obtain the current window; when the volatility is in the third fluctuation range, use the preset window as the current window.

[0033] Specifically, when the volatility of the indicator is below 10%, the statistical window is expanded (e.g., from 10 minutes to 1 hour); when the volatility exceeds 30%, the window is automatically shortened to quickly respond to sudden load changes; when the volatility is in other ranges, the window size is not changed.

[0034] 4) Retrieve the historical business load curves of the aforementioned indicators and perform dynamic identification to obtain the current load status.

[0035] 5) Match the corresponding deviation threshold according to the current load status and the current deviation judgment rule, and mark it as the current threshold.

[0036] This application provides a threshold adaptive mechanism based on load intervals: it dynamically identifies the current load interval (peak period, off-peak period, and low period, which correspond to different load states) based on the historical business load curve, and applies different deviation rate thresholds to different intervals. For example: During peak periods (TPS greater than 1.5 times the baseline value), the deviation rate threshold is set at 40%.

[0037] During off-peak periods (TPS is between 0.7 and 1.5 times the baseline value), the deviation rate threshold is set at 30%.

[0038] During the trough period (TPS is 0.7 times lower than the baseline value), the deviation rate threshold is set at 20%.

[0039] By identifying the current load status in real time, the deviation tolerance is automatically adjusted to the corresponding threshold, thereby reducing false alarms during low load periods and improving detection sensitivity during high load periods.

[0040] The aforementioned automated adjustment mechanism can improve adaptability in dynamic scenarios.

[0041] (22) Calculate the baseline value based on the current window. This application supports multiple time windows (such as configurable windows such as 5 minutes and 1 hour) to calculate the historical mean and variance of each monitoring indicator as the baseline value. In addition, this application also supports independent statistics based on the type of indicator and monitoring dimension (link / component).

[0042] In practical applications, this application employs a sliding window incremental update mechanism: the baseline value is updated in real time using a sliding window method, enabling the baseline to dynamically track the changing trends of the system's operating status. The incremental update calculation method is: baseline value_new= α ×current value + (1-α)×baseline value_old; α is the smoothing coefficient, which can be configured.

[0043] (23) Based on the current deviation judgment rule, calculate the deviation data according to the baseline value and the current value of the indicator.

[0044] For example, the current deviation judgment rule is to calculate the deviation rate between the current value of the indicator and the baseline value in real time when using a deviation rate threshold. The calculation formula is: Deviation rate = |current value - baseline value| / baseline value 100%.

[0045] This application supports trend anomaly detection, such as calculating the growth rate of an indicator (the current value relative to the average growth rate over a past time window). When the growth rate exceeds a threshold, it is determined to be a trend anomaly, and the deviation rate and trend status of each indicator are output. Different deviation judgment rules can be used for different indicators.

[0046] (24) Based on the deviation data and the current threshold, the indicator is judged to be abnormal, and when the judgment result is that there is an abnormality, the indicator is marked as an abnormal indicator.

[0047] (25) Mark the abnormal indicator and its corresponding deviation data and timestamp as an indicator abnormal event; multiple indicator abnormal events constitute an indicator abnormal event set.

[0048] If step 202 is considered as a dynamic baseline calculation module, the input is a standardized link / component indicator dataset, and the output is the baseline value of each indicator, the dynamic deviation rate, and the anomaly judgment result. The dynamic baseline calculation module includes a threshold adaptive update unit, a historical baseline statistics unit, and a dynamic deviation calculation unit. The threshold adaptive update unit executes step (21), the historical baseline statistics unit executes step (22), and the dynamic deviation calculation unit executes steps (23)-(25), while also taking into account the output function. Through the above processing, this application can achieve dynamic adaptive adjustment under different business cycles and load fluctuations, realize a smooth evolution from static threshold to intelligent threshold, and significantly improve the stability and accuracy of anomaly detection.

[0049] Step 203: Based on the preset link-component mapping table and combined with the set of abnormal events of the indicators, determine the abnormal events associated with the link and the component.

[0050] In a specific application, step 203 includes the following steps: (31) For any indicator abnormal event in the set of indicator abnormal events, when the indicator abnormal event occurs at a link node, the corresponding processing component is matched according to the preset link-component mapping table, and deep indicator collection is triggered to determine the corresponding component status and the first associated timestamp; the location of the link node where the indicator abnormal event occurs, the corresponding component status and the first associated timestamp constitute the link-component associated abnormal event.

[0051] In the deep metrics collection process, deep metrics refer to the collection of fine-grained internal resource and performance metrics for abnormal objects to assist in root cause analysis and causal verification. Examples include: 1) For message relay cache nodes, collecting cache hit rate, I / O wait latency, queue lock wait time, and thread blocking rate; 2) For core business processing components, collecting JVM heap / non-heap memory utilization, GC (Garbage Collection) duration, object creation rate, and thread state distribution; 3) For communication service nodes, collecting network round-trip time (RTT), connection retry rate, and average interface response time; 4) For databases or caching middleware, collecting lock wait time, slow query ratio, and index hit rate.

[0052] (32) When the abnormal event of the indicator occurs in the component layer, the corresponding link node is matched according to the preset link-component mapping table, and the data backtracking collection process is triggered to determine the corresponding abnormal link location and the second associated timestamp; the abnormal link location, the component layer status in which the abnormal event of the indicator occurs and the second associated timestamp constitute the link-component associated abnormal event.

[0053] Specifically, after triggering, bidirectional tracing is performed based on the link-component mapping relationship: when a link node triggers an anomaly (such as cache queue backlog), a deep metric collection task will be triggered down to the corresponding component layer to determine whether there is a resource bottleneck or performance degradation, thereby determining whether the link anomaly is caused by a component problem; when a component layer triggers an anomaly (such as JVM heap memory overflow in a business processing component), the affected link nodes are queried in reverse to determine whether the anomaly causes a decrease in link throughput or an increase in message latency. The tracing results will form "associated anomaly events," including the location of the anomaly link, the corresponding component status, and the associated timestamp.

[0054] If step 203 is considered a two-dimensional linkage analysis module, then the inputs are: a set of abnormal indicator events, a link-component mapping relationship table, and raw indicator data (input as needed). The output is: associated abnormal events, including the link abnormality location, component status, and abnormal association time. Additionally, this module may include a link-component association unit and an abnormal bidirectional tracing unit.

[0055] The link-component association unit pre-stores a mapping table between link nodes and their corresponding processing components, i.e., a preset link-component mapping table, used to establish the correspondence between link-level and component-level metrics. Examples of mapping relationships include: message relay cache queue node ↔ core business processing component, communication service ↔ network I / O component, etc. Through the above mapping settings, when an anomaly in a link-level metric is detected, the corresponding component-level object can be immediately located, achieving a forward association from link to component; conversely, when an anomaly in a component-level metric is detected, it can be traced back to the affected link node, achieving a reverse analysis of the impact of the component on the link.

[0056] The bidirectional anomaly tracing unit includes the steps (31)-(32) above. When an abnormal indicator is detected in either the link or component, a bidirectional linkage analysis process will be executed, including two functions: in-depth indicator collection and correlation tracing analysis. In addition, during the anomaly verification stage, the original indicator samples before standardization will be referenced again for: anomaly playback and time accuracy verification to determine the actual trigger time of the anomaly; fitting analysis of the anomaly curve to identify the aomaly mutation point; and extraction of the original features of performance parameters in in-depth collection to improve the analysis accuracy.

[0057] Through a bidirectional anomaly correlation mechanism between links and components and a deep data acquisition mechanism, this application can complete cross-dimensional and cross-time correlation location and mutual verification analysis, and realize automated fault tracing and dual-dimensional linkage judgment.

[0058] Step 204: Based on the component topology dependency relationship, according to the abnormal events associated with the link and component, perform multi-source abnormal event fusion processing, alarm processing and root cause identification, and determine the aggregated alarm event and root cause location report.

[0059] In one specific application, step 204 includes: (41) Based on the abnormal events associated with the links and components, determine the abnormal indicators and corresponding deviation data and duration in the associated abnormal events.

[0060] (42) Based on the component topology dependency relationship, determine the influence range and propagation direction of abnormal indicators in the associated abnormal events.

[0061] (43) Determine the alarm level based on the deviation data, the duration, and the scope of influence; specifically, automatically trigger the alarm level according to the following rules: S1 (Critical Alarm): Triggered when the deviation rate exceeds 50% and the abnormal duration exceeds 5 minutes, or when both the link and component are abnormal.

[0062] S2 (Intermediate Alarm): Triggered when the deviation rate is between 30% and 50% and the abnormality of a single-layer component lasts for more than 3 minutes.

[0063] S3 (Minor Alarm): Triggered when the deviation rate is below 30% but a trend anomaly occurs three or more times consecutively.

[0064] The alarm level determined by the above rules will determine the scope of subsequent push and alarm suppression strategies.

[0065] (44) Based on the abnormal indicators and the corresponding alarm levels, perform alarm redundancy suppression, alarm deduplication and alarm buffering to determine the aggregated alarm events.

[0066] Alarm redundancy suppression and alarm deduplication: When an underlying component (such as a cache or database) triggers an S1-level alarm, the alarm push of its upstream dependent components (such as business processing components and management backend) under the same cause type is suspended to prevent duplicate reporting. For alarms that are triggered consecutively for the same metric within a short period of time, a deduplication and merging operation is performed, retaining only the latest alarm status, the first trigger time, and the cumulative duration.

[0067] Alarm buffering and stability strategy: For short-term fluctuating indicators (such as TPS jitter and network latency), a buffer parameter N (configurable, default 3 times) is set. A formal alarm is only generated when the indicator is judged as abnormal N times consecutively; if it is a single or intermittent abnormality, the trigger is delayed or ignored directly. This mechanism effectively suppresses false alarms caused by short-term jitter.

[0068] (45) For multiple related abnormal events, based on the component topology dependencies, time sequence, and event correlation, determine the root cause component and generate a root cause localization report. Specifically, this step includes the following processing flow: 1) Based on the associated timestamps corresponding to the anomaly indicators, sort multiple associated anomaly events by time to identify the first component layer or link node to experience an anomaly. That is, statistically analyze the occurrence time sequence of all associated anomaly events and prioritize identifying the first component or link node to experience an anomaly.

[0069] 2) Based on the component topology dependencies, perform topology path verification on multiple associated anomaly events and eliminate component layers or link nodes without dependencies to obtain multiple candidate root cause points. Specifically, confirm the upstream and downstream dependency paths between anomaly nodes in the topology dependencies to ensure that the time-preceding anomaly is located in the downstream source direction of the topology (starting component). If no dependency exists, exclude the node from the root cause candidate status.

[0070] 3) Perform correlation analysis on multiple related anomaly events and filter them according to preset correlation values ​​to obtain multiple related candidate root causes. Specifically, perform correlation analysis on the changing trends of indicators among related anomaly events, calculate the Pearson correlation coefficient or time-series cross-correlation value, and consider a significant correlation when the correlation coefficient is higher than 0.7.

[0071] 4) The root cause component is determined by combining the first component layer or link node where the anomaly occurs, multiple candidate root cause points, and multiple related candidate root cause points. This application uses "first occurrence + existence of topology path + significant correlation of indicators" as the three conditions for determining the root cause component. If a component meets all of the above conditions, it is determined to be the root cause component.

[0072] 5) Generate a root cause localization report based on the root cause components and their corresponding influence range and propagation direction.

[0073] If step 204 is considered as an alarm aggregation root cause analysis module, the inputs are: abnormal events associated with links and components, and a component topology dependency table; the outputs are: aggregated alarm events and root cause location reports (including root cause components, scope of impact, and propagation path). The alarm aggregation root cause analysis module includes a topology dependency management unit, an alarm suppression unit, and a time-series correlation analysis unit.

[0074] The topology dependency management unit is used to maintain the component topology and dependencies of the orchestration system, and provides a dependency path query interface for the alarm analysis phase. In other words, this unit stores and maintains the component topology dependencies. Specifically, the topology dependencies are stored in the form of configuration tables or graphs to describe the dependency order of components at each layer, for example: basic security layer cache components → core capability layer business components → business association layer management backend. This topology dependency is dynamically loaded at runtime to determine the direction and scope of anomaly propagation, providing a structural basis for subsequent alarm aggregation and root cause localization.

[0075] The alarm suppression unit can execute the above steps (41)-(44). This unit receives associated abnormal events and performs multi-level alarm aggregation, deduplication and suppression logic based on three indicators: the severity of the abnormality (i.e., deviation data), duration and scope of impact. The time-series correlation analysis unit can execute the above step (45) and perform automatic determination analysis of the root cause component by combining the temporal sequence relationship, topological dependency relationship and indicator correlation.

[0076] Through the above-mentioned multi-layer aggregation and correlation analysis mechanism, the fusion processing of multi-source abnormal events, automatic suppression of alarm storms and automatic identification of root causes can be realized, significantly improving the alarm signal-to-noise ratio and operation and maintenance location efficiency.

[0077] In one specific application, the method of this application also includes: Step 205: Based on the aggregated alarm events and the root cause localization report, perform tiered push and notification distribution. This step can be performed by the alarm push unit, which performs tiered push and notification distribution according to the alarm level (S1 / S2 / S3) and scope of responsibility based on the aggregated alarm events and the root cause localization report. Its processing flow includes: (51) Formatting of alarm information: The aggregated alarm events are standardized into a unified structure, including fields such as alarm ID, level identifier, trigger time, duration, root cause component identifier, scope of impact and abnormal description; the unified format ensures that the alarm can be recognized and called by subsequent systems (message notification, operation and maintenance center or automated troubleshooting platform).

[0078] (52) Tiered push and grouping strategy: Differentiated push is performed based on alarm level and responsibility area: S1 level alarms (critical alarms): Immediately and synchronously pushed to the operations and maintenance team and system architecture manager, triggering voice calls or instant message notifications; S2 level alarms (intermediate alarms): Pushed to the corresponding subsystem operations and maintenance and technical support personnel, and notified via email and messaging systems; S3 level alarms (minor alarms): Recorded in the alarm event log, only displayed as a reminder in the visual interface. Push strategies can be configured according to organizational structure, enabling grouping and distribution by role, region, and service scope.

[0079] Step 206 involves presenting a multi-dimensional visualization of each link node in the orchestration system, the link indicator dataset, each component layer, the component indicator dataset, the aggregated alarm events, and the root cause analysis report. This step can be performed by the visualization unit, which is responsible for presenting a multi-dimensional visualization of all link and component monitoring data, abnormal alarm information, and root cause analysis results within the orchestration system, enabling interactive operation and maintenance support. Its main functions are as follows: (61) Dual-dimensional monitoring and display: The system simultaneously displays the operational status of the link dimension and the component dimension. The link section displays real-time indicator curves from the client to each backend node (communication service, message broker, cache, business component); the component section displays resource utilization, processing performance and health score. Both use graphical identifiers (such as color and flashing status) to represent abnormal nodes, achieving end-to-end full path visualization.

[0080] (62) Display of related abnormal relationships: When an abnormal event occurs, an “abnormal propagation path diagram” is automatically drawn, displaying the root cause component of the abnormality, the affected link nodes and the propagation path; for example, the fault link of “core business component → message relay cache → management backend” can be dynamically highlighted on the interface.

[0081] (63) Alarm trend and statistical analysis: Statistical aggregation of historical alarms to generate time series trend charts, including indicators such as the number of alarms, alarm level distribution, average recovery time (MTTR) and false alarm rate, to provide reference for subsequent operation and maintenance optimization.

[0082] (64) Correlation time sequence backtracking function: Operation and maintenance personnel can select any time window through the interface to view the evolution trajectory of indicators before and after the occurrence of abnormal events (such as TPS decline curve, CPU utilization change curve), and intuitively analyze the fault tracing process.

[0083] The alarm push unit and visualization unit can be integrated into the alarm push and visualization module. This module takes aggregated alarm events and root cause reports as input and outputs tiered alarm notifications, visualized reports, and closed-loop feedback data, achieving an automated closed loop from anomaly detection to presentation and intervention. Operations personnel can obtain real-time status of links and components, anomaly causes, propagation paths, and processing feedback information from a unified interface, significantly improving alarm response efficiency and problem localization accuracy.

[0084] Step 207: Receive feedback or remarks from operations and maintenance personnel, associate them with the aggregated alarm event, and back them up. This step can also be performed by the alarm push unit. Specifically, after operations and maintenance personnel handle the alarm, they can confirm the receipt or make remarks through the monitoring system interface. The alarm push unit will record the handler, processing time, and recovery result in the alarm database, forming closed-loop event data. This data will then be used to optimize the dynamic baseline and threshold model, achieving self-learning and parameter tuning.

[0085] In a specific application, the steps of this application's method can be simply described as the following process: Step S1, Data Acquisition: Acquire link / component metrics.

[0086] The system collects metrics such as queue backlog and processing rate of client, communication service, message broker, message relay cache, and business processing components; the component-layered collection unit collects resource and business metrics of the business association layer, core capability layer, and basic support layer; the metric preprocessing subunit standardizes the above data and outputs a structured dataset.

[0087] Taking the link anomaly caused by the performance degradation of the message relay cache as an example, the collected message queue length index of the message relay cache configuration class was 2000, and the cache queue TPS was 50 (the normal baseline TPS is 100); the collected message consumption rate of the core business processing component was 40, and the JVM heap memory utilization rate was 95%; the above data were standardized to output a structured index set.

[0088] In terms of data acquisition, agentless acquisition methods (such as eBPF technology) can be used instead of deploying acquisition agents, thereby reducing system resource consumption.

[0089] Step S2, Dynamic baseline calculation: Calculate the deviation rate and calculate anomalies.

[0090] The configured time window (e.g., 1 hour) is used to calculate the baseline value of the indicator; the deviation rate between the current value and the baseline value is calculated; if the deviation rate exceeds the configurable threshold (e.g., >30%), the indicator is judged to be abnormal.

[0091] Corresponding to the example above of link anomalies caused by message relay cache performance degradation, the baseline value of the cache queue TPS for 1 hour is calculated to be 100. The deviation rate is calculated as |50-100| / 100×100%=50%>30%, which is judged as an anomaly. Finally, the abnormal event of the output indicator is: abnormal drop in message relay cache queue TPS, with a deviation rate of 50%.

[0092] In addition, the baseline calculation method can also use machine learning algorithms (such as LSTM) to predict the normal range of indicators, replacing the baseline calculation method of "historical mean + deviation rate" to achieve more accurate anomaly detection.

[0093] Step S3, dual-dimensional linkage analysis: correlation link anomalies and component status.

[0094] Based on the pre-stored mapping relationship, match the processing component corresponding to the abnormal link node; for the abnormal link node, trigger the deep indicator collection of the corresponding component, or for the abnormal component, trace the link node it affects and generate related abnormal events.

[0095] Corresponding to the example above where message relay cache performance degradation caused a link anomaly, the processing component corresponding to the message relay cache queue was matched to the core business processing component. This triggered a deep metric collection of the core business processing component, which revealed that its JVM heap memory utilization was 95%. It was determined that the core business processing component's memory overflow caused a decrease in consumption rate, which in turn led to the backlog of the message relay cache queue. Finally, the associated abnormal event was output: Message relay cache queue backlog (link anomaly) ← Core business processing component JVM memory overflow (component anomaly).

[0096] Step S4, Alarm Aggregation and Root Cause Analysis: Suppress redundant alarms and locate the root cause.

[0097] Load component topology dependencies, suppress redundant upstream alarms caused by abnormal underlying components, and perform alarm deduplication and buffering; compare the occurrence time of abnormal events, determine the root cause component, and generate a root cause location report.

[0098] Corresponding to the example above where message relay cache performance degradation caused a chain anomaly, the loading dependency relationship is: message relay cache → core business processing component → management backend. The core business processing component was detected as the underlying abnormal component, so the "operation success rate decline" alarm in the management backend was paused to avoid an alarm storm. Comparing timestamps, the core business processing component's memory overflow occurred earlier than the message relay cache queue backlog, determining the JVM memory overflow in the core business processing component as the root cause. Finally, a root cause location report was output: the root cause component is the core business processing component, the affected chain node is the message relay cache, the affected scope is the management backend, and the propagation path is core business processing component → message relay cache → management backend.

[0099] In addition, alarm aggregation can be processed by using graph neural networks (GNNs) to analyze component topology relationships, replacing the root cause localization method of "temporal correlation + dependency tracing" and improving the accuracy of root cause identification in complex topologies.

[0100] Step S5, Alarm Output and Visualization: Generate handling suggestions based on root causes.

[0101] Alarms are pushed to operations and maintenance personnel according to their alarm levels, and dual-dimensional monitoring data and root cause analysis results are displayed.

[0102] Corresponding to the example above of link anomalies caused by message relay cache performance degradation, S2-level alarms are pushed to the operation and maintenance personnel of core business processing components, and the correlation between link backlog and component memory, as well as the root cause analysis results, are displayed on the visualization interface.

[0103] Step S6, Operation and Maintenance Intervention.

[0104] Operations and maintenance personnel will take targeted measures against the root cause components based on the root cause localization report (such as cache performance optimization or business component restart).

[0105] In response to the example above where message relay cache performance degradation caused link anomalies, the operations and maintenance personnel optimized the JVM parameters of the core business processing components, released memory, restored the consumption rate of the core business processing components to 100, reduced the backlog of the message relay cache queue to 0, and the system returned to normal.

[0106] This application provides a bidirectional correlation mapping mechanism between link monitoring and component hierarchical monitoring, enabling bidirectional tracing of abnormal events between links and components. It also addresses the poor adaptability of static thresholds by using a dynamic deviation rate calculation method based on historical baselines. Furthermore, it determines the source of failure through temporal correlation analysis based on alarm suppression and root cause localization logic rooted in component topology dependencies. The method and corresponding orchestration system of this application can be applied to fields such as intelligent robot control and autonomous vehicles in the Internet of Things (IoT).

[0107] Based on the same inventive concept, this application also provides a system. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more system embodiments provided below can be found in the limitations of the method above, and will not be repeated here.

[0108] In one exemplary embodiment, an alarm root cause localization system for an orchestration system is provided, comprising: The dual-dimensional monitoring data acquisition module is used to collect link indicator datasets and component indicator datasets from the orchestration system.

[0109] Anomaly detection module (corresponding) Figure 3 The dynamic baseline calculation module is used to load the current threshold adaptation mechanism and, based on the current threshold adaptation mechanism, perform anomaly identification according to the link indicator dataset and the component indicator dataset to determine the set of indicator anomaly events.

[0110] The dual-dimensional linkage analysis module is used to determine the abnormal events associated with links and components based on a preset link-component mapping table and the set of abnormal events of the indicators.

[0111] The alarm aggregation root cause analysis module is used to perform multi-source abnormal event fusion processing, alarm processing, and root cause identification based on the component topology dependency relationship and the abnormal events associated with the link and component, and to determine the aggregated alarm events and root cause location report.

[0112] like Figure 3 As shown, this application also includes an alarm output and visualization module. Furthermore, the dual-dimensional monitoring data acquisition module includes a message link acquisition unit, a component-level acquisition unit, and an indicator preprocessing unit; the dynamic baseline calculation module includes a threshold adaptive update unit, a historical baseline statistics unit, and a dynamic deviation calculation unit; the dual-dimensional linkage analysis module includes a link-component association unit and an anomaly bidirectional tracing unit; the alarm aggregation root cause analysis module includes a topology dependency management unit, an alarm suppression unit, and a time-series correlation analysis unit; and the alarm output and visualization module includes an alarm push unit and a visualization display unit. The specific functions of each unit are detailed in the method section above and will not be repeated here.

[0113] Compared with the prior art, this application has the following advantages: (1) Achieve dual-dimensional linkage positioning and improve fault diagnosis efficiency: Through the link-component association mechanism, the fault location time is shortened from the traditional minute level to the second level, and the positioning accuracy is improved to more than 95%.

[0114] (2) Dynamic thresholds adapt to business load and reduce false alarm rate: Compared with static threshold alarms, the false alarm rate is reduced by 60%, avoiding the interference of invalid alarms during off-peak periods on operation and maintenance.

[0115] (3) Suppress alarm storms and improve alarm signal-to-noise ratio: Through the topology-dependent alarm suppression strategy, the number of alarms is reduced by 70%, and maintenance personnel can focus on core faults.

[0116] (4) End-to-end monitoring coverage to eliminate monitoring blind spots: realize full-link monitoring coverage of message chain and full-level component, and increase the monitoring coverage rate from the original 60% to 100%.

[0117] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an alarm root cause localization method for the orchestration system.

[0118] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] In one exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the above-described method embodiments.

[0120] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0121] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0124] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for locating the root cause of alarms in an orchestration system, characterized in that, The method includes: Collect link metric datasets and component metric datasets from the orchestration system; Load the current threshold adaptive mechanism, and based on the current threshold adaptive mechanism, perform anomaly identification according to the link indicator dataset and the component indicator dataset to determine the set of indicator anomaly events; Based on the preset link-component mapping table and combined with the set of abnormal events of the indicators, abnormal events associated with links and components are determined; Based on the component topology dependency, and according to the abnormal events associated with the links and components, multi-source abnormal events are fused, alarms are processed, and root causes are identified to determine aggregated alarm events and root cause location reports.

2. The alarm root cause localization method for the orchestration system according to claim 1, characterized in that, Collect link metric datasets and component metric datasets from the orchestration system, including: Collect raw link indicator data of each link node in the orchestration system; the link node is one of the following: client, communication service, message broker, message relay cache, and business processing component; Collect raw component indicator data for each layer of the orchestration system; the component layer is one of the following: business association layer, core capability layer, and basic support layer; All the original link indicator data and the original component indicator data are standardized to obtain the link indicator dataset and the component indicator dataset.

3. The alarm root cause localization method for the orchestration system according to claim 1, characterized in that, Based on the current threshold adaptive mechanism, anomaly identification is performed according to the link indicator dataset and the component indicator dataset to determine the set of indicator anomaly events, including: For any metric in the link metric dataset and the component metric dataset, the current threshold, current window, and current deviation judgment rule are determined based on the current threshold adaptive mechanism. Calculate the baseline value based on the current window; Based on the current deviation judgment rule, the deviation data is calculated according to the baseline value and the current value of the indicator; Based on the deviation data and the current threshold, the indicator is judged to be abnormal, and if the judgment result is that there is an anomaly, the indicator is marked as an abnormal indicator; The abnormal indicator, its corresponding deviation data, and timestamp are marked as an indicator abnormal event; multiple indicator abnormal events constitute an indicator abnormal event set.

4. The alarm root cause localization method for the orchestration system according to claim 3, characterized in that, Based on the current threshold adaptive mechanism, the current threshold, current window, and current deviation judgment rule are determined, including: The corresponding deviation judgment rule is invoked based on the type of the indicator, and used as the current deviation judgment rule; Retrieve the time series of the aforementioned indicator, and perform moving variance analysis and cycle identification to determine volatility; When the volatility is within the first fluctuation range, the preset window is increased to obtain the current window; when the volatility is within the second fluctuation range, the preset window is decreased to obtain the current window; when the volatility is within the third fluctuation range, the preset window is used as the current window. Retrieve the historical business load curves of the aforementioned indicators and perform dynamic identification to obtain the current load status; Based on the current load status and the current deviation judgment rule, a corresponding deviation threshold is matched and marked as the current threshold.

5. The alarm root cause localization method for the orchestration system according to claim 1, characterized in that, Based on the preset link-component mapping table and combined with the set of abnormal events for the indicators, abnormal events associated with links and components are determined, including: For any metric abnormal event in the set of metric abnormal events, when the metric abnormal event occurs at a link node, the corresponding processing component is matched according to the preset link-component mapping table, and deep metric collection is triggered to determine the corresponding component status and the first associated timestamp; the location of the link node where the metric abnormal event occurs, the corresponding component status, and the first associated timestamp constitute a link-component associated abnormal event. When the abnormal event of the indicator occurs in the component layer, the corresponding link node is matched according to the preset link-component mapping table, and the data backtracking collection process is triggered to determine the corresponding abnormal link location and the second associated timestamp; the abnormal link location, the component layer status in which the abnormal event of the indicator occurs, and the second associated timestamp constitute the link-component associated abnormal event.

6. The alarm root cause localization method for the orchestration system according to claim 1, characterized in that, Based on component topology dependencies, and according to the abnormal events associated with the links and components, multi-source abnormal events are fused, alarms are processed, and root causes are identified to determine aggregated alarm events and root cause location reports, including: Based on the abnormal events associated with the links and components, determine the abnormal indicators, corresponding deviation data, and duration of the associated abnormal events; Based on the component topology dependencies, determine the scope of influence and propagation direction of abnormal indicators in associated abnormal events; The alarm level is determined based on the deviation data, the duration, and the scope of impact. Based on the aforementioned abnormal indicators and corresponding alarm levels, alarm redundancy suppression, alarm deduplication, and alarm buffering are performed to determine aggregated alarm events. For multiple related abnormal events, the root cause component is determined and a root cause localization report is generated based on the component topology dependency, time sequence and event correlation.

7. The alarm root cause localization method for an orchestration system according to claim 6, characterized in that, For multiple related anomaly events, based on the component topology dependencies, time sequence, and event correlation, the root cause component is determined and a root cause localization report is generated, including: Based on the associated timestamps of the abnormal indicators, multiple associated abnormal events are sorted by time to identify the first component layer or link node where the abnormality occurred. Based on the component topology dependencies, topology path verification is performed on multiple associated abnormal events, and component layers or link nodes without dependencies are eliminated to obtain multiple candidate root cause points. Correlation analysis was performed on multiple related abnormal events, and the results were filtered based on preset correlation values ​​to obtain multiple related candidate root causes. By combining the first component layer or link node where the anomaly occurred, multiple candidate root cause points, and multiple related candidate root cause points, the root cause component is determined. Based on the root cause components and their corresponding influence range and propagation direction, a root cause localization report is generated.

8. The alarm root cause localization method for the orchestration system according to claim 1, characterized in that, The method further includes: Based on the aggregated alarm events and the root cause localization report, perform tiered push and notification distribution; The orchestration system includes each link node, the link indicator dataset, each component layer, the component indicator dataset, the aggregated alarm events, and the root cause localization report, which are presented in a multi-dimensional visualization. Receive feedback or remarks from maintenance personnel, associate them with the aggregated alarm events, and back them up.

9. An alarm root cause localization system for an orchestration system, characterized in that, The system includes: The dual-dimensional monitoring data acquisition module is used to collect link indicator datasets and component indicator datasets in the orchestration system; An anomaly identification module is used to load the current threshold adaptive mechanism and, based on the current threshold adaptive mechanism, perform anomaly identification according to the link indicator dataset and the component indicator dataset to determine the set of indicator anomaly events. The dual-dimensional linkage analysis module is used to determine the abnormal events associated with links and components based on a preset link-component mapping table and the set of abnormal events of the indicators. The alarm aggregation root cause analysis module is used to perform multi-source abnormal event fusion processing, alarm processing, and root cause identification based on the component topology dependency relationship and the abnormal events associated with the link and component, and to determine the aggregated alarm events and root cause location report.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the alarm root cause localization method of the orchestration system according to any one of claims 1-8.

Images