A cross-platform big data resource monitoring and visualization analysis system

Abstract

This invention discloses a cross-platform big data resource monitoring and visualization analysis system, relating to the field of computer data processing technology. It includes: a cross-platform data acquisition and standardization module, used to collect performance indicator data and structured log data in real time from heterogeneous data sources consisting of physical servers, virtualization platforms, cloud environments, container clusters, and distributed applications. Based on a preset unified data model, the module cleans, converts, and timestamps the collected raw data, outputting a standardized data stream. This invention solves the problem of data fusion difficulties across multiple platforms by integrating heterogeneous data sources and outputting a unified standardized data stream through the cross-platform data acquisition and standardization module; and it automatically identifies monitored entities and causal relationships using a dynamic causal graph construction and update module, constructing and continuously updating a dynamic directed graph, thus overcoming the deficiency of existing systems in presenting dynamic causal relationships between indicators.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, specifically to a cross-platform big data resource monitoring and visualization analysis system. Background Technology

[0002] With the rapid development of information technology, big data technology has become a core driving force supporting modern enterprise decision-making and operations. As a key component of enterprise data infrastructure, big data resource monitoring and visualization analysis systems aim to provide managers with a comprehensive understanding of system performance and resource utilization by collecting, processing, and displaying massive amounts of operational data in real time, thereby ensuring business continuity and optimizing resource allocation.

[0003] Cross-platform big data resource monitoring and visualization analysis systems are a key area of ​​current technological development. These systems need to integrate data sources from physical servers, virtualization platforms, cloud environments, container clusters, and various distributed applications. Their core objective is to build a unified, real-time data view to reveal the resource status and business operation patterns in complex heterogeneous environments.

[0004] Existing technologies typically rely on deploying agents on various platforms or utilizing standard interfaces to collect performance metrics and log data, and then visualizing them through a unified dashboard. However, when facing highly dynamic and elastically scalable cloud-native environments, existing systems have the following limitations in their visualization and analysis capabilities: The system cannot automatically identify and intuitively present the causal relationships between dynamic changes in different resource metrics. Current visualization charts mostly remain at the level of displaying independent metrics such as CPU utilization, memory usage, and network throughput side-by-side or simply overlaying them. When a microservice or application component experiences performance fluctuations, operations personnel can only rely on personal experience, manually comparing the fluctuations of multiple time-series curves to infer the root cause, such as determining whether the query pressure of the underlying database caused application-layer response latency, or whether the elastic scaling operation of a certain service caused instantaneous network bandwidth congestion. This method of relying on manual experience for correlation analysis is difficult to quickly and accurately locate the transmission path and root cause nodes of faults in a distributed environment with a large number of monitoring metrics, complex inter-service call relationships, and rapid changes. This leads to low efficiency in problem diagnosis and response, and fails to meet the urgent need for intelligent operations and maintenance for real-time visualization and automated analysis of root causes. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a cross-platform big data resource monitoring and visualization analysis system, which solves the problems of existing technologies being unable to automatically present dynamic causal relationships of indicators and relying on manual correlation analysis, resulting in inefficient fault location.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a cross-platform big data resource monitoring and visualization analysis system, comprising: The cross-platform data acquisition and standardization module is used to collect performance index data and structured log data in real time from heterogeneous data sources consisting of physical servers, virtualization platforms, cloud environments, container clusters and distributed applications. Based on a preset unified data model, it cleans, converts the format and aligns the timestamps of the collected raw data, and outputs a standardized data stream. The dynamic causal graph construction and update module is used to automatically construct and continuously update a dynamic directed graph representing the causal relationship between monitored entities based on the standardized data stream. The dynamic causal graph construction and update module includes a monitored entity abstraction layer and a causal relationship discovery engine. The monitored entity abstraction layer identifies and abstracts monitored entities from the standardized data according to preset entity type definition rules. The causal relationship discovery engine calculates the transfer entropy and mutual information values ​​between the indicator sequences of any two monitored entities in a preset sliding time window, and determines the existence and direction of the causal edge based on the comparison result of the transfer entropy value and a preset threshold, and assigns a weight value to the causal edge. The multidimensional root cause visualization engine is used to receive the dynamic directed graph and real-time performance index data, and generate a composite visualization interface that integrates topology view, heat map and causal flow view. The automated root cause analysis and decision-making module is used to monitor abnormal fluctuations in performance index data. When an abnormal event is detected, it traverses the causal edges in the dynamic directed graph in reverse to collect candidate root cause nodes. Based on the abnormal index magnitude of the candidate nodes, the causal path weight to the initial abnormal node, and the historical root cause frequency, it calculates a comprehensive root cause score and outputs a sorted list of suspected root cause nodes and the corresponding propagation path sequence.

[0007] Preferably, the cross-platform data acquisition and standardization module has multiple built-in platform adapters. The platform adapters acquire raw monitoring data by calling the standard management interface of the target platform or by deploying lightweight agentless probes. The performance index data includes CPU utilization, memory usage, disk I / O operation frequency, network bandwidth utilization, number of container replicas, and service request response time. The structured log data includes application error logs, service call chain tracing identifiers, and system event records.

[0008] Preferably, the monitoring entity types identified by the monitoring entity abstraction layer include host entities, virtual machine entities, container entities, microservice entities, database entities, and message queue entities; each monitoring entity is assigned a globally unique identifier and associated with its platform type, physical location, and the sequence of real-time performance metrics collected.

[0009] Preferably, when determining a causal edge, the causal relationship discovery engine determines that there exists a causal edge from the first monitoring entity to the second monitoring entity if and only if the transfer entropy value from the first indicator of the first monitoring entity to the second indicator of the second monitoring entity exceeds a preset threshold, and the transfer entropy value is greater than the reverse transfer entropy value from the second indicator to the first indicator; the weight value of the causal edge is calculated by weighting the transfer entropy value and the mutual information value.

[0010] Preferably, the dynamic causal graph construction and update module further includes a graph structure manager; the graph structure manager constructs a system-wide causal graph using the monitored entity as nodes and the causal edges output by the causal relationship discovery engine as directed edges; the graph structure manager continuously receives causal edge update events output by the causal relationship discovery engine in each time window and incrementally updates the system-wide causal graph; the causal edge update events include the addition of new causal edges, the update of the weights of existing causal edges, and the deletion of old causal edges.

[0011] Preferably, the graph structure manager also maintains a timeliness queue for each causal edge, recording its weight change history within the most recent N time windows; the graph structure manager implements an edge decay mechanism, for a causal edge that has not been updated within multiple consecutive time windows, its weight value is reduced by a preset decay rate, and when the weight value decays to below the deletion threshold, the causal edge is removed from the causal graph of the entire system.

[0012] Preferably, the multidimensional root cause visualization engine includes a topology view renderer, a heatmap matrix generator, and a causal flow view synthesizer; the topology view renderer maps the nodes and edges in the dynamic directed graph to visualization elements, the nodes are rendered using different geometric icons according to their monitored entity types, the size of the nodes is dynamically scaled according to their real-time performance indicators, and the thickness and color depth of the directed edges are mapped according to their weight values.

[0013] Preferably, the heatmap matrix generator obtains a set of key performance indicators from the central time-series database for the most recent M sampling times, constructs a two-dimensional data matrix with monitored entities as rows and time points as columns, and maps the value of each cell in the matrix to a color to generate a heatmap that changes over time; the heatmap is linked with the nodes in the topology view, and when the user selects a node in the topology view, the heatmap highlights the trajectory of the indicator changes of the entity corresponding to that node at all time points.

[0014] Preferably, the causal flow view synthesizer extracts a downstream causal path subgraph starting from the suspected root cause node from the dynamic directed graph, rearranges the subgraph using a force-directed layout algorithm, and starts the animation rendering process; the animation rendering process simulates a pulse effect, which starts from the suspected root cause node and lights up the downstream nodes on the path sequentially along the direction of the causal edge according to the propagation delay determined by the edge weight.

[0015] Preferably, the automated root cause analysis decision module determines abnormal events by judging whether the current value of any performance indicator exceeds a preset multiple of the standard deviation of its dynamic baseline threshold; the comprehensive root cause score is calculated as follows: the comprehensive root cause score equals the abnormal amplitude of the candidate node's own indicator multiplied by a first weighting coefficient, plus the average weight of all edges on the causal path from the candidate node to the initial abnormal node within the most recent time window multiplied by a second weighting coefficient, plus the frequency of the candidate node as a root cause in historical abnormal events multiplied by a third weighting coefficient; the number K of the suspected root cause node list is dynamically determined based on the total number of currently monitored entities.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention integrates heterogeneous data sources through a cross-platform data acquisition and standardization module, outputting a unified and standardized data stream to solve the problem of data fusion difficulties across multiple platforms. It automatically identifies monitored entities and causal relationships using a dynamic cause-effect graph construction and update module, building and continuously updating a dynamic directed graph to overcome the shortcomings of existing systems in presenting dynamic causal relationships between indicators. A multi-dimensional root cause visualization engine provides an integrated multi-view composite interface, allowing operations and maintenance personnel to intuitively grasp system status and causal propagation. An automated root cause analysis and decision-making module quickly detects anomalies and locates suspected root causes and propagation paths, replacing manual correlation analysis, improving fault diagnosis and response efficiency, meeting the needs of intelligent operations and maintenance, and effectively ensuring business continuity. Attached Figure Description

[0017] Figure 1 This is a system structure diagram of the present invention; Figure 2 This is a flowchart of the dynamic cause-effect graph construction and updating process of the present invention; Figure 3 This is a flowchart of the automated root cause analysis decision-making process of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1

[0020] Please see Figure 1-3 This embodiment provides a cross-platform big data resource monitoring and visualization analysis system, and describes it in detail in conjunction with an enterprise hybrid cloud operation and maintenance scenario. The system includes: The cross-platform data acquisition and standardization module is used to collect performance index data and structured log data in real time from heterogeneous data sources consisting of physical servers, virtualization platforms, cloud environments, container clusters and distributed applications. Based on a preset unified data model, it cleans, converts, and aligns the collected raw data with timestamps, and outputs a standardized data stream.

[0021] Specifically, the cross-platform data acquisition and standardization module incorporates multiple platform adapters, employing adaptive acquisition methods for different target platforms. For platforms supporting standard management interfaces, raw monitoring data is obtained by calling their provided RESTful APIs or SNMP interfaces. For scenarios that do not support standard interfaces or have strict requirements for acquisition performance, a lightweight agentless probe is deployed. This probe consumes very few system resources and can collect data in real time without affecting business operations. The collected performance metrics cover key dimensions such as CPU utilization, memory usage, disk I / O operation frequency, network bandwidth utilization, number of container replicas, and service request response time. Structured log data includes core information related to troubleshooting, such as application error logs, service call chain tracing identifiers, and system event records. The collected raw data had issues such as inconsistent formats, inconsistent timestamps, and noisy data. Therefore, the raw data was processed according to a pre-defined unified data model. During the data cleaning process, a statistical outlier removal method was used to identify and remove data that deviated from the data mean by more than three standard deviations. At the same time, missing values ​​were filled. Linear interpolation was used to fill missing numerical data, and mode filling was used to fill missing character data. During the format conversion process, the heterogeneous data formats output from different platforms were uniformly converted to JSON format to ensure the consistency of the data structure. The timestamp alignment operation used the system's unified UTC time as the benchmark to convert the timestamps from different collection sources into a unified format, and the time precision was uniformly adjusted to the millisecond level. Finally, a standardized data stream was output, laying the foundation for subsequent causal relationship analysis and visualization.

[0022] In one specific embodiment, the dynamic causal graph construction and update module is used to automatically construct and continuously update a dynamic directed graph representing the causal relationship between monitored entities based on standardized data streams. The dynamic causal graph construction and update module includes a monitoring entity abstraction layer and a causal relationship discovery engine. The monitoring entity abstraction layer identifies and abstracts monitored entities from standardized data according to preset entity type definition rules. The causal relationship discovery engine calculates the transfer entropy and mutual information values ​​between the indicator sequences of any two monitored entities in a preset sliding time window, and determines the existence and direction of the causal edge based on the comparison result of the transfer entropy value and a preset threshold, and assigns a weight value to the causal edge.

[0023] Specifically, the dynamic cause-effect graph construction and update module includes a monitoring entity abstraction layer that identifies and abstracts monitoring entities from standardized data based on preset entity type definition rules. Monitoring entity types include host entities, virtual machine entities, container entities, microservice entities, database entities, and message queue entities. Each monitoring entity is assigned a globally unique identifier generated using the UUID v4 algorithm to ensure no duplication across the entire system. This identifier is also associated with its platform type, physical location, and the collected real-time performance metric sequence, making the attribute information of each monitoring entity complete and traceable. The cause-effect discovery engine performs causal relationship analysis using a preset sliding time window. The size of the sliding time window is set according to the data collection frequency. For example, when the data collection frequency is once per second, the sliding time window is set to 60 seconds, meaning that each analysis considers the metric sequence data within the most recent 60 seconds. This setting balances the real-time nature and accuracy of the analysis, ensuring timely capture of recent causal relationship changes while avoiding insufficient data volume due to an excessively small window, which could negatively impact the analysis results. Within each time window, calculate the transfer entropy and mutual information values ​​between the indicator sequences of any two monitored entities. Transfer entropy measures the causal direction and strength between two random variables, and its calculation formula is as follows: ; in, This represents the transfer entropy value from the indicator sequence of monitored entity X to the indicator sequence of monitored entity Y. Represents conditional entropy. This represents the index value of the monitored entity Y at time t+1. This represents the sequence of indicators for the monitored entity Y over the k time steps prior to time t. This represents the sequence of indicators for monitored entity X over the m time intervals prior to time t. The values ​​of k and m are determined based on the temporal correlation of the indicators, and are typically set to 30, meaning that historical data from the most recent 30 time intervals are referenced. The mutual information value is used to measure the degree of correlation between two indicator sequences, and the calculation formula is: ; in, This represents the mutual information value between the indicator sequences of monitored entity X and the indicator sequences of monitored entity Y. and Let X and Y represent the marginal entropy, respectively. This represents the joint entropy of X and Y. When determining causal edges, the causal relationship discovery engine determines the existence of a causal edge from the first monitoring entity to the second monitoring entity if and only if the transfer entropy value from the first indicator of the first monitoring entity to the second indicator of the second monitoring entity exceeds a preset threshold, and this transfer entropy value is greater than the reverse transfer entropy value from the second indicator to the first indicator. The transfer entropy threshold is set using a statistical method, based on the distribution of transfer entropy values ​​under historical normal operating conditions, taking the 95th percentile as the initial threshold, and subsequently dynamically adjusted according to changes in system operating conditions to ensure that the threshold can adapt to the causal relationship determination needs under different scenarios. The weight value of the causal edge is calculated by the weighted sum of the transfer entropy value and the mutual information value, using the following formula: ; in, This represents the weight value of the causal edge. and These are the weighting coefficients. Set to 0.6. The value is set to 0.4. This setting is based on the core role of propagation entropy in determining causal direction and the auxiliary role of mutual information in measuring correlation strength. It has been verified through multiple experiments and can accurately reflect the strength of causal relationships.

[0024] The dynamic causal graph construction and update module also includes a graph structure manager. The graph structure manager uses monitored entities as nodes and causal edges output by the causal relationship discovery engine as directed edges to construct a system-wide causal graph, enabling the causal relationships between entities in the system to be represented graphically. The graph structure manager continuously receives causal edge update events output by the causal relationship discovery engine in each time window, including the addition of new causal edges, updates to the weights of existing causal edges, and deletion of old causal edges. It then performs incremental updates to the system-wide causal graph. This incremental update method avoids the resource consumption and time delays associated with full updates, ensuring that the causal graph can reflect the dynamic changes in causal relationships in the system in real time. Simultaneously, the graph structure manager maintains a timeliness queue for each causal edge, recording its weight change history over the most recent N time windows. The value of N is determined based on the system's dynamic change frequency; for example, setting it to 10 records the weight changes over the most recent 10 time windows, facilitating the tracing of the evolution of causal relationships. The graph structure manager also implements an edge decay mechanism. For a causal edge that has not been updated within multiple consecutive time windows, its weight value is reduced by a preset decay rate, which is set to decrease by 10% each time window. When the weight value decays to below the deletion threshold, the causal edge is removed from the causal graph of the entire system. The deletion threshold is set to 30% of the initial threshold. Through this edge decay mechanism, causal relationships that no longer exist in the system can be cleaned up in a timely manner, avoiding redundancy in the causal graph structure, ensuring the simplicity and effectiveness of the causal graph, and allowing subsequent root cause analysis to focus on the currently valid causal relationships.

[0025] In one specific embodiment, a multidimensional root cause visualization engine is used to receive dynamic directed graphs and real-time performance index data, and generate a composite visualization interface that integrates topology views, heatmaps, and causal flow views.

[0026] Specifically, the multidimensional root cause visualization engine receives dynamic directed graphs and real-time performance metric data, generating a composite visualization interface that integrates topology views, heatmaps, and causal flow views. This interface can intuitively display the system's resource status and causal relationships from multiple dimensions, solving the problem that existing visualization charts can only display individual metrics independently and cannot present the dynamic causal relationships between metrics. The topology view renderer included in the multidimensional root cause visualization engine maps nodes and edges in the dynamic directed graph to visualization elements. Nodes are rendered using different geometric icons based on their monitored entity type. For example, host entities use rectangular icons, virtual machine entities use circular icons, container entities use triangular icons, microservice entities use diamond icons, database entities use trapezoidal icons, and message queue entities use pentagonal icons, enabling different types of entities to be quickly distinguished. The size of the nodes is dynamically scaled according to their real-time performance metrics, using a linear mapping method. The normal range of performance metrics corresponds to the baseline node size. When the metric value exceeds the normal range, the node is enlarged proportionally. The further the metric value deviates from the normal range, the larger the node becomes, allowing operations personnel to quickly identify entities with abnormal performance. The thickness and color depth of directed edges are mapped according to their weight values. The larger the weight value, the thicker the directed edge and the darker the color, which intuitively reflects the strength of the causal relationship and helps maintenance personnel focus on key causal connections.

[0027] The heatmap matrix generator retrieves data on key performance indicators (KPIs) from a central time-series database over the most recent M sampling times. The value of M is determined based on monitoring requirements; for example, setting it to 1440 means retrieving sampling data for each minute within the last 24 hours. It constructs a two-dimensional data matrix with monitored entities as rows and time points as columns, mapping the value of each cell in the matrix to a color using a gradient color scheme—lower values ​​correspond to lighter colors, and higher values ​​to darker colors. This generates a heatmap that changes over time, making the performance indicator trends of each monitored entity clearly visible at different time points. The heatmap is linked to nodes in the topology view. When a user selects a node in the topology view, the heatmap highlights the corresponding entity's KPI trajectory across all time points, facilitating in-depth analysis of the entity's performance history and quickly pinpointing time intervals of performance anomalies.

[0028] The causal flow view synthesizer extracts downstream causal path subgraphs starting from suspected root cause nodes from a dynamic directed graph. A force-directed layout algorithm is then used to rearrange these subgraphs. This algorithm simulates attraction and repulsion in the physical world, ensuring a uniform distribution of nodes and preventing overlap while maintaining clear connections between causal edges. An animation rendering process is then initiated, simulating a pulse effect. Starting from the suspected root cause node, the pulse effect illuminates downstream nodes along the causal edge, according to the propagation delay determined by the edge weight. The propagation delay is inversely proportional to the edge weight; a larger edge weight results in a smaller propagation delay and a faster pulse propagation speed. This visually demonstrates the propagation process of anomalies from the root cause node to downstream nodes, allowing maintenance personnel to clearly understand the path and scope of anomaly propagation and providing clear guidance for troubleshooting.

[0029] In one specific embodiment, the automated root cause analysis decision module is used to monitor abnormal fluctuations in performance index data. When an abnormal event is detected, it traverses the dynamic directed graph in reverse along the causal edges to collect candidate root cause nodes. Based on the abnormal index magnitude of the candidate nodes, the causal path weight to the initial abnormal node, and the historical root cause frequency, it calculates a comprehensive root cause score and outputs a sorted list of suspected root cause nodes and the corresponding propagation path sequence.

[0030] Specifically, the automated root cause analysis decision module monitors abnormal fluctuations in performance indicator data. This module determines abnormal events by judging whether the current value of any performance indicator exceeds a preset multiple of the standard deviation of its dynamic baseline threshold. The dynamic baseline threshold is calculated using the sliding window mean method, and the calculation formula is as follows: ; in, This represents the dynamic baseline threshold at time t. This indicates the length of the sliding window, set to 100 sampling periods. This represents the performance index value for the i-th sampling period. The preset standard deviation is set to 2, meaning that when the current value of the performance index is... satisfy When this occurs, it is determined to be an abnormal event, among which The standard deviation of the performance index values ​​over the most recent L sampling periods is represented by the following formula: ; This anomaly detection method can adapt to dynamic changes in performance indicators and avoid false alarms and false negatives caused by fixed thresholds. When an anomaly is detected, the dynamic directed graph is traversed in reverse along the causal edges to collect candidate root cause nodes. The traversal process starts from the initial anomaly node and follows the reverse direction of the causal edges to collect all upstream nodes that may have caused the anomaly as candidate root cause nodes, ensuring that no potential root cause is missed.

[0031] The comprehensive root cause score is calculated based on the magnitude of the anomaly in the candidate node's indicators, the causal path weights to the initial anomaly node, and the historical root cause frequency. The formula for calculating the comprehensive root cause score is as follows: ; in, This represents the overall root cause score. This indicates the abnormal magnitude of the candidate node's own indicators. , The current value of the candidate node indicator. The dynamic baseline threshold for candidate node indicators. The standard deviation of the candidate node index; This represents the average weight of all edges on the causal path from the candidate node to the initial anomalous node within the most recent time window. , Let be the number of edges on the causal path. Let be the weight value of the i-th edge; This indicates the frequency with which the candidate node was a root cause in historical anomalous events. , This represents the number of times the candidate node has been identified as the root cause in historical anomalies. This represents the total number of historical anomalies. , , The first, second, and third weighting coefficients are defined using the analytic hierarchy process (AHP). A judgment matrix is ​​constructed, and a consistency check is performed to determine the final weighting coefficients. , , This weighting setting comprehensively considers the severity of the indicator anomaly, the strength of the causal relationship, and historical experience, accurately measuring the likelihood of candidate nodes as root causes. The automated root cause analysis and decision-making module outputs a sorted list of suspected root cause nodes and their corresponding propagation path sequences. The number K of the suspected root cause node list is dynamically determined based on the total number of currently monitored entities, set to 5% of the total number of currently monitored entities, with a minimum of 3 and a maximum of 10, ensuring that the output root cause list is both concise and comprehensive, facilitating maintenance personnel to quickly focus on key root causes and improving fault diagnosis efficiency.

[0032] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0033] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cross-platform big data resource monitoring and visual analysis system, characterized in that, include: The cross-platform data acquisition and standardization module is used to collect performance index data and structured log data in real time from heterogeneous data sources consisting of physical servers, virtualization platforms, cloud environments, container clusters and distributed applications. Based on a preset unified data model, it cleans, converts the format and aligns the timestamps of the collected raw data, and outputs a standardized data stream. The dynamic cause-effect graph construction and update module is used to automatically construct and continuously update a dynamic directed graph representing the causal relationship between monitored entities based on the standardized data stream. The dynamic causal graph construction and update module includes a monitoring entity abstraction layer and a causal relationship discovery engine. The monitoring entity abstraction layer identifies and abstracts monitoring entities from the standardized data according to preset entity type definition rules. The causal relationship discovery engine calculates the transfer entropy and mutual information values ​​between the indicator sequences of any two monitoring entities using a preset sliding time window as the unit, and determines the existence and direction of the causal edge based on the comparison result of the transfer entropy value and a preset threshold, and assigns a weight value to the causal edge. The multidimensional root cause visualization engine is used to receive the dynamic directed graph and real-time performance index data, and generate a composite visualization interface that integrates topology view, heat map and causal flow view. The automated root cause analysis and decision-making module is used to monitor abnormal fluctuations in performance index data. When an abnormal event is detected, it traverses the causal edges in the dynamic directed graph in reverse to collect candidate root cause nodes. Based on the abnormal index magnitude of the candidate nodes, the causal path weight to the initial abnormal node, and the historical root cause frequency, it calculates a comprehensive root cause score and outputs a sorted list of suspected root cause nodes and the corresponding propagation path sequence. 2.The cross-platform big data resource monitoring and visual analysis system of claim 1, wherein, The cross-platform data acquisition and standardization module has multiple built-in platform adapters. These platform adapters acquire raw monitoring data by calling the target platform's standard management interface or deploying lightweight agentless probes. The performance metrics include CPU utilization, memory usage, disk I / O operation frequency, network bandwidth utilization, number of container replicas, and service request response time. The structured log data includes application error logs, service call chain trace identifiers, and system event records.

3. The cross-platform big data resource monitoring and visualized analysis system of claim 1, wherein, The monitoring entity abstraction layer identifies monitoring entity types including host entities, virtual machine entities, container entities, microservice entities, database entities, and message queue entities; each monitoring entity is assigned a globally unique identifier and associated with its platform type, physical location, and the sequence of real-time performance metrics collected.

4. The cross-platform big data resource monitoring and visualization analysis system according to claim 1, characterized in that, When determining a causal edge, the causal relationship discovery engine determines that there exists a causal edge from the first monitoring entity to the second monitoring entity if and only if the transfer entropy value from the first indicator of the first monitoring entity to the second indicator of the second monitoring entity exceeds a preset threshold, and the transfer entropy value is greater than the reverse transfer entropy value from the second indicator to the first indicator; the weight value of the causal edge is calculated by weighting the transfer entropy value and the mutual information value.

5. The cross-platform big data resource monitoring and visualization analysis system according to claim 1, characterized in that, The dynamic cause-effect graph construction and update module also includes a graph structure manager; the graph structure manager constructs a system-wide cause-effect graph using the monitored entity as a node and the causal edges output by the cause-effect discovery engine as directed edges; the graph structure manager continuously receives the causal edge update events output by the cause-effect discovery engine in each time window and performs incremental updates on the system-wide cause-effect graph. The causal edge update events include the addition of new causal edges, the updating of the weights of existing causal edges, and the deletion of old causal edges.

6. The cross-platform big data resource monitoring and visualization analysis system according to claim 5, characterized in that, The graph structure manager also maintains a timeliness queue for each causal edge, recording its weight change history within the most recent N time windows; the graph structure manager implements an edge decay mechanism, for a causal edge that has not been updated within multiple consecutive time windows, its weight value is reduced by a preset decay rate, and when the weight value decays to below the deletion threshold, the causal edge is removed from the causal graph of the entire system.

7. The cross-platform big data resource monitoring and visualization analysis system according to claim 1, characterized in that, The multidimensional root cause visualization engine includes a topology view renderer, a heatmap matrix generator, and a causal flow view synthesizer. The topology view renderer maps the nodes and edges in the dynamic directed graph to visualization elements. Nodes are rendered using different geometric icons based on their monitored entity types. The size of the nodes is dynamically scaled according to their real-time performance indicators. The thickness and color depth of the directed edges are mapped according to their weight values.

8. The cross-platform big data resource monitoring and visualization analysis system according to claim 7, characterized in that, The heatmap matrix generator obtains a set of key performance indicators from the central time-series database for the most recent M sampling times. It constructs a two-dimensional data matrix with the monitored entities as rows and time points as columns, and maps the value of each cell in the matrix to a color to generate a heatmap that changes over time. The heatmap is linked with the nodes in the topology view. When the user selects a node in the topology view, the heatmap highlights the trajectory of the indicator changes of the entity corresponding to that node at all time points.

9. The cross-platform big data resource monitoring and visualization analysis system according to claim 7, characterized in that, The causal flow view synthesizer extracts the downstream causal path subgraph starting from the suspected root cause node from the dynamic directed graph, rearranges the subgraph using a force-directed layout algorithm, and starts the animation rendering process. The animation rendering process simulates a pulse effect, which starts from the suspected root cause node and lights up the downstream nodes along the causal edge in sequence according to the propagation delay determined by the edge weight.

10. A cross-platform big data resource monitoring and visualization analysis system according to claim 1, characterized in that, The automated root cause analysis decision module determines abnormal events by judging whether the current value of any performance indicator exceeds a preset multiple of the standard deviation of its dynamic baseline threshold. The formula for calculating the comprehensive root cause score is as follows: the comprehensive root cause score equals the abnormal magnitude of the candidate node's own index multiplied by the first weight coefficient, plus the average weight of all edges on the causal path from the candidate node to the initial abnormal node within the most recent time window multiplied by the second weight coefficient, plus the frequency of the candidate node as a root cause in historical abnormal events multiplied by the third weight coefficient; the number K of the suspected root cause node list is dynamically determined based on the total number of currently monitored entities.

Images