Visual monitoring system and method for local large model cluster resource running state and data flow direction

Abstract

The invention relates to the technical field of system monitoring, and discloses a visual monitoring system and method for a local large model cluster resource operation state and a data flow direction, and the method comprises the steps that a data collection layer integrates multi-dimensional operation data full-amount collection; the data transmission layer adopts Kafka to construct a high throughput distributed message queue, and performs parallel data processing through a partition mechanism; the data processing layer realizes real-time aggregation calculation and anomaly detection based on Flink; the storage layer adopts a multi-modal architecture, and uses a multi-dimensional database to store resource indexes, log link data and dynamic relationships respectively; the service layer triggers an abnormal event through a rule engine; the application layer adopts a force-oriented layout algorithm to render a dynamic topological graph and supports node drilling and instrument panel display; the technical problem that in the prior art, full-link and real-time monitoring cannot be carried out on the running state of system cluster resources and the data flow direction is solved.

Description

Technical Field

[0001] This invention relates to the field of system monitoring technology, specifically to a visualization monitoring system and method for the operating status and data flow of local large-scale model cluster resources. Background Technology

[0002] Traditional monitoring systems focus on the isolated display of hardware resource metrics, lacking a metric system strongly correlated with large-scale business models. This prevents operations and maintenance personnel from directly identifying business performance bottlenecks through the monitoring system, requiring them to rely on manual experience to locate problems.

[0003] Traditional monitoring systems often scatter resource monitoring data, job logs, network traffic, and model version information across independent systems, creating significant data silos. Troubleshooting requires cross-system, multi-dimensional data analysis, which is inefficient and prone to missing crucial clues due to manual intervention.

[0004] Meanwhile, monitoring visualization often uses static graphs, which cannot intuitively display the dynamic data flow and real-time load relationships between cluster nodes. In distributed training scenarios, the dynamic relationships between nodes need to be analyzed by manually drawing topology diagrams, which has poor real-time performance and makes it difficult to track the entire data flow chain. Summary of the Invention

[0005] The present invention aims to provide a visualization monitoring system and method for the operation status and data flow of local large model cluster resources, so as to solve the technical problem that the existing technology cannot perform full-link, real-time monitoring of the operation status and data flow of system cluster resources.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a visualization monitoring system for the operating status and data flow of local large-scale model cluster resources, comprising: The data acquisition layer integrates hardware resource probes, application link tracers, job log collectors, and network traffic probes to collect multi-dimensional runtime data from various nodes in the cluster, which is used to capture all east-west and north-south traffic. The data transmission layer uses Kafka to build a high-throughput distributed message queue and processes the collected data in parallel through a partitioning mechanism. The data processing layer is based on Flink to achieve real-time aggregation computing and anomaly detection; it adopts a multimodal storage architecture, using a time-series database to store resource indicator data, Elasticsearch to store logs and tracing data, and Neo4j graph database to store the dynamic relationships between nodes, services, and tasks. The service layer combines stream processing results with historical data and uses a rule engine to enable real-time triggering of abnormal events; it also provides a unified interface for aggregating multi-source data. At the application layer, a force-directed layout algorithm is used to render a global dynamic topology map in real time, supporting node drilling to view resource details, and dynamically displaying resource utilization through a custom dashboard.

[0007] The principle and advantages of this solution are as follows: This solution constructs an integrated end-to-end architecture of "collection-transmission-processing-storage-visualization". Through multi-dimensional data collection, high-throughput Kafka message queues, Flink real-time computing, multi-modal storage, and force-directed layout dynamic topology visualization, it achieves end-to-end, real-time, and intelligent monitoring of the local large-scale model cluster resource operation status and data flow. The system triggers abnormal event alarms through a rule engine and provides a unified interface to aggregate multi-source data, ultimately displaying the cluster status intuitively in the form of dynamic topology maps and custom dashboards.

[0008] Breaking down data silos in traditional monitoring tools, this system unifies the collection, storage, correlation, and display of four types of data: metrics, logs, and network traffic, thus resolving the problem of fragmented monitoring dimensions. Based on the Flink stream processing engine, it achieves dynamic threshold alarms and anomaly detection, surpassing traditional static threshold polling and reducing false alarms and missed alarms. Employing a force-directed layout algorithm, it renders a global dynamic topology graph in real time, intuitively displaying node status, network connections, and data flow, and supporting node drill-down to view resource details, thus addressing the pain points of static visualization.

[0009] Preferably, as an improvement, in the data acquisition layer, the hardware resource probe collects the utilization rates of CPU, memory, disk, network hardware, and container resources through Prometheus NodeExporter; the application link tracer tracks the distributed task call chain through the Jaeger probe; the job log collector collects logs through Fluentd and stores them in Elasticsearch; and the network traffic probe captures inter-node traffic through sFlow and performs quintuple analysis.

[0010] Preferably, as an improvement, the data acquisition layer adopts a dynamic sampling rate adjustment mechanism, which dynamically adjusts the sampling frequency based on the real-time load of the nodes. The calculation formula is as follows: ; Where S(t) is the sampling interval at the current time t, This is the upper limit of the sampling interval. Let L(t) be the lower limit of the sampling interval, and L(t) be the resource utilization rate of the processing unit at the current time t. σ is the load threshold, k is the adjustment coefficient, and σ is the smoothing factor.

[0011] The beneficial effects of this improvement are: the sampling interval is dynamically adjusted through a nonlinear function, increasing the sampling frequency to capture key details under high load and reducing the frequency to reduce resource consumption under low load, thus achieving a balance between monitoring accuracy and resource consumption.

[0012] Preferably, as an improvement, the data acquisition layer performs edge computing preprocessing on the acquired raw data. By deploying lightweight Flink tasks on the node side, the acquired raw data is initially aggregated to reduce the amount of data transmitted to Kafka.

[0013] The benefits of this improvement are: deploying lightweight Flink tasks on the node side for initial aggregation, such as a 10-second window sliding average, reduces the amount of data transmitted to Kafka, lowers network bandwidth requirements, and improves the real-time performance of data processing.

[0014] Preferably, as an improvement, the number of Kafka partitions in the data transmission layer is automatically adjusted based on the consumption speed. When the consumer group's processing speed is lower than the production speed, Kafka automatically increases the number of partitions and triggers a rebalance mechanism to reallocate partitions.

[0015] The benefits of this improvement are: when the consumer group's processing speed is lower than the production speed, the number of partitions is automatically increased and a rebalance mechanism is triggered to achieve traffic peak shaving and valley filling, support real-time transmission of PB-level data with zero loss, and solve the data transmission bottleneck problem.

[0016] Preferably, as an improvement, the time-series data in the storage layer is stored in InfluxDB hot data and S3 cold data by time partitioning, with the most recent 7 days of data stored in InfluxDB, and data older than 7 days archived to S3 and batch analyzed through Athena.

[0017] The benefits of this improvement are: data stored in InfluxDB for the most recent 7 days supports nanosecond-level queries, while data older than 7 days is archived to S3 for batch analysis via Athena, reducing storage costs while meeting the needs for fast queries of hot data and long-term analysis of cold data.

[0018] Preferably, as an improvement, the Kafka producer introduces a token bucket algorithm to limit burst traffic, presets a maximum number of messages produced per second, and when the traffic exceeds the preset maximum number of messages produced per second, it is buffered and sent smoothly to avoid Broker overload.

[0019] The benefits of this improvement are: it presets the maximum number of messages produced per second, such as 100,000; if this number is exceeded, it buffers and sends messages smoothly, avoiding Broker overload and ensuring stable data transmission in high-concurrency scenarios.

[0020] A method for visually monitoring the operational status and data flow of local large-scale model cluster resources includes: Step 1: Collect multi-dimensional runtime data from each node of the cluster by integrating hardware resource probes, application link tracers, job log collectors, and network traffic probes; Step 2: Use Kafka to build a high-throughput distributed message queue, and process data in parallel through multiple partitions to achieve traffic peak shaving and valley filling; Step 3: Implement real-time aggregation calculation and anomaly detection based on Flink, supporting dynamic configuration of threshold alarm rules; Step 4: Combine the stream processing results with historical data to trigger anomaly event alerts through the rule engine; Step 5: Use the force-directed layout algorithm to render the global dynamic topology map in real time, displaying node status, network connections and data flow.

[0021] The beneficial effects of this improvement are: through end-to-end optimization—data collection → transmission → processing → storage → visualization—it achieves an upgrade in the operation and maintenance model from passive response to proactive optimization. Multi-dimensional data collection solves the problem of monitoring being disconnected from business operations, real-time processing and alarms solve the problem of alarm lag, and dynamic visualization solves the problem of unintuitive relationships between nodes, ultimately forming a closed-loop intelligent monitoring system. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the structure of an embodiment of the present invention. Detailed Implementation

[0023] The following detailed description illustrates the specific implementation method: Example A visualization monitoring system and method for the operational status and data flow of local large-scale model cluster resources, the basics of which are shown in the attached figure. Figure 1 As shown, a visualization monitoring system for the operational status and data flow of a local large-scale model cluster includes: The data acquisition layer integrates hardware resource probes, application link tracers, job log collectors, and network traffic probes to collect multi-dimensional runtime data from various nodes in the cluster, achieving full capture of east-west and north-south traffic. East-west traffic originates from within the local network, while north-south traffic originates from outside the local network.

[0024] The page probe automatically marks the time, status and path of task modules, supporting the tracking of the entire lifecycle from task start to completion.

[0025] Hardware resource probes collect the utilization of hardware and container resources such as CPU, memory, disk, and network through deployment tools. Application tracing automatically tracks the call chain of distributed tasks through probes integrated into the application page, recording the time, status, and path of each module.

[0026] Job log collection involves deploying logging tools to collect log files from various tasks and systems for analyzing progress and error information. Network traffic probes utilize technology or network device traffic monitoring functions to collect and analyze east-west and north-south network traffic to map data transmission paths.

[0027] Specifically, hardware resource probes can collect data every 5 seconds using Prometheus Node Exporter deployed on each node, including CPU utilization (e.g., GPU-0 utilization of 85% on node A), memory usage (e.g., 128GB of remaining memory on node B), disk I / O (e.g., SSD read / write speed of 500MB / s on node C), and network bandwidth (e.g., uplink traffic of 2Gbps on node D). Simultaneously, cAdvisor monitors container-level resource usage, such as the memory usage of Docker containers for training tasks.

[0028] The application link tracer can inject Jaeger probes into various modules of model training, such as data loading, forward propagation, back propagation, and parameter updates, to automatically trace the call chain. For example, when the "data loading" module takes longer than a preset threshold, such as 200ms, the probe will mark the module as a "potential bottleneck" and record its call path, such as reading data from HDFS → data preprocessing → input to the model layer.

[0029] The job log collector gathers training task log files through Fluentd. For example, TensorFlow logs contain progress information such as "epoch 1 / 100, loss=0.23" and error logs such as "CUDA out of memory". These logs are parsed and stored in Elasticsearch, supporting full-text search, such as searching for the keyword "loss" to view all loss value records.

[0030] Network traffic probes: sFlow probes deployed on switches capture east-west traffic (such as 10Gbps traffic for inter-node model parameter synchronization) and north-south traffic (such as 500Mbps traffic for loading training data from external storage systems). The probes perform 5-tuple analysis on the traffic, including source IP, destination IP, protocol, port, and timestamp, and label the traffic type, such as RPC calls and file transfers.

[0031] During data acquisition, a dynamic sampling rate adjustment mechanism is employed, adjusting the sampling frequency based on the real-time load of the nodes. When the real-time load exceeds the load threshold, the sampling frequency is adjusted using the following formula: ; Where S(t) is the adjusted sampling interval at the current time t, and S(t) is adjusted by adding a lower limit to the sampling interval through a dynamic adjustment term. The dynamic adjustment term is obtained by mapping the load L(t) to the adjustment range of the sampling interval through the nonlinear function tanh. This is the upper limit of the sampling interval, i.e., the minimum sampling frequency, used to prevent the system from overloading due to excessively small sampling intervals. The sampling interval is the lower limit, i.e., the maximum sampling frequency, used to ensure that critical details can still be captured under high load. L(t) is the processing unit resource utilization rate at the current time t, which covers general computing resources such as CPU, GPU, TPU, and DSP; it is obtained through actual monitored load indicators. σ is the load threshold, which serves as the benchmark for triggering adjustment. When the load exceeds this value, the sampling interval is shortened. k is the adjustment coefficient, used to control the sensitivity to load changes; a larger k value results in more aggressive adjustment. σ is the smoothing factor, used to suppress sampling interval oscillations caused by instantaneous load fluctuations.

[0032] The nonlinear mapping part, i.e. The hyperbolic tangent function will reduce the load deviation. Normalized to [ The interval is [1, 1].

[0033] Dynamic adjustment term, i.e. By combining k and tanh, the sampling interval is made to be within Smooth changes within the range, avoiding step-like abrupt changes.

[0034] For example, consider an increase in CPU utilization from 50% to 95%. Set the lower limit of the sampling interval. The upper limit of the sampling interval is 2 seconds. Set the time to 10 seconds and set the load threshold. The initial value is 75%. The adjustment coefficient k is 0.8, and the smoothing factor σ is 5%. Therefore, when the initial time L(t) is 50%, the sampling interval is 12s.

[0035] When the load rises to 95%: Nonlinear mapping part The dynamic adjustment term S(t) = 10 × (1 (0.8 × 1) + 2 = 4s. The sampling interval is shortened from 10 seconds to 4 seconds, which meets the requirement of increasing the sampling frequency under high load.

[0036] k, σ, It can be dynamically adjusted according to business needs, such as increasing k during peak periods to enhance response speed.

[0037] The collected raw data undergoes edge computing preprocessing. By deploying lightweight Flink tasks on the node side, preliminary aggregation of the collected raw data is performed, such as moving average of processing unit resource utilization. For example, a 10-second window moving average is calculated for the GPU-0 utilization of node A to reduce the amount of data transmitted to Kafka.

[0038] The data transmission layer employs Kafka to build a high-throughput distributed message queue, achieving traffic smoothing, decoupling production and consumption, and ensuring zero data loss under high-load scenarios. A partitioning mechanism is used to smooth out traffic peaks and valleys, supporting real-time transmission of petabyte-scale data.

[0039] All collected data is pushed to a Kafka cluster. Kafka processes data in parallel using three partitions, each corresponding to a consumer group. For example, hardware resource data is written to the "metrics-topic" partition, application tracing data to the "tracing-topic" partition, and log data to the "logs-topic" partition. Kafka's ISR mechanism ensures zero data loss; even if one broker fails, other brokers can take over the partition. By adjusting the batch size and compression algorithm (such as LZ4), the system can support real-time transmission of petabytes of data, with a peak throughput of 1 GB / s.

[0040] The number of partitions is automatically adjusted based on the consumption speed of Kafka partitions. For example, when the processing speed of the consumer group for the "metrics-topic" is lower than the production speed, Kafka automatically increases the number of partitions to 5 and triggers the rebalance mechanism to reallocate the partitions.

[0041] Introduce a token bucket algorithm on the Kafka producer side to limit bursts of traffic. For example, set the maximum number of messages produced per second to 100,000; if this is exceeded, buffer and send messages smoothly to avoid broker overload.

[0042] The data processing layer, based on Flink, implements real-time aggregation computation and anomaly detection, supporting dynamic configuration of threshold alarm rules. It consumes data from Kafka, performs real-time computation and persistent storage, and uses real-time data streams for real-time aggregation, computation, and anomaly detection; for example, it determines whether a threshold is exceeded based on the computation results.

[0043] It supports multimodal storage, using a time-series database to store timestamped resource metric data, suitable for efficient querying and trend chart visualization. It uses Elasticsearch's log database to store logs and tracing data, leveraging its powerful full-text search capabilities for quick problem localization. Neo4j's graph database stores and computes the dynamic relationships between nodes, services, and tasks, used to generate and display real-time topology maps and data transmission paths.

[0044] During real-time computation in Flink, Flink consumer groups read data streams from Kafka and perform real-time aggregation computations. For example: A sliding window is used to calculate the resource utilization of the processing unit, such as a window size of 1 minute and a sliding step of 30 seconds, to generate an average utilization curve per minute.

[0045] Anomaly detection is performed on network traffic. The Z-Score algorithm is used to identify sudden increases in traffic, such as a sudden surge in traffic between nodes from 1Gbps to 5Gbps, which is then marked as "traffic anomaly".

[0046] The task time is statistically analyzed to generate the data needed for a Gantt chart, such as task start time, end time, and duration.

[0047] In multimodal storage, time-series databases, such as InfluxDB, store resource metrics data and support queries by time range, such as querying the CPU utilization trend over the past hour.

[0048] Elasticsearch stores logs and tracing data, supporting full-text search and aggregate queries, such as counting the number of "error logs".

[0049] The Neo4j graph database stores relationships between nodes, services, and tasks, such as the dependency relationship "node A → task T1 → service S1", and supports the generation of real-time topology graphs, such as using Cypher to query "find all nodes directly connected to task T1".

[0050] Time-series data is partitioned by time and stored in InfluxDB (hot data) and S3 (cold data). For example, CPU utilization data for the most recent 7 days is stored in InfluxDB, supporting nanosecond-level queries; data older than 7 days is archived to S3 and batch analyzed using Athena, reducing storage costs.

[0051] The service layer combines stream processing results with historical data, and uses a rules engine to trigger abnormal events in real time, supporting alarm grading, blocking, and confirmation mechanisms. It provides a unified interface to aggregate data from multiple sources, supporting dynamic invocation by front-end visualization components.

[0052] Based on the results of real-time computation using Flink or periodically queried stored data, judgments are made according to predefined anomaly threshold rules. Once triggered, an alarm event is immediately generated and sent to the notification center via an API. A unified API interface is provided to the frontend to aggregate data from different databases, avoiding direct access to multiple data sources by the frontend.

[0053] Based on Flink's computation results, the rule engine dynamically configures threshold-based alarm rules. For example: When the CPU utilization of a node exceeds 90% for 5 consecutive minutes, a "high load alarm" is triggered and marked as "P1 level", which is the highest priority.

[0054] When abnormal network traffic persists for more than 1 minute, a "traffic anomaly alarm" is triggered, and related nodes and tasks are associated with it.

[0055] It supports alarm masking, such as preventing known nighttime maintenance operations from triggering alarms and alarm confirmations, and preventing repeated notifications of alarms that have been confirmed to have been handled by maintenance personnel.

[0056] The alarm notification center sends alarm information through multiple channels. For example: P1 level alarms are simultaneously notified to the operations and maintenance team leader via WeChat, SMS, and email.

[0057] P2 level alerts, such as memory usage exceeding 80%, will only notify the relevant engineers via email.

[0058] The alarm information includes detailed context, such as node IP, task ID, and abnormal metric values, and supports integration with external systems via API, such as automatically creating Jira fault tickets.

[0059] At the application layer, a force-directed layout algorithm is used to render a global dynamic topology graph in real time. The global dynamic topology graph includes cluster node status, network connections, and data flow, and supports drilling down on nodes to view resource details.

[0060] Customizable dashboards support drag-and-drop components (charts / dashboards / tables) for dynamic display of resource utilization, with multi-dimensional filtering and comparison. Task Gantt charts visually represent task dependencies and execution progress, supporting critical path analysis and bottleneck identification.

[0061] The data traceability view, based on task ID or file identifier, graphically traces the entire lifecycle and flow path, meeting the audit traceability requirements.

[0062] It integrates historical alarm query, active alarm handling, and alarm rule configuration functions, supporting multi-channel notifications such as email, SMS, and WeChat Work. Alarm information is notified to operations and maintenance personnel through multiple channels including DingTalk, SMS, email, or internal enterprise IM. An API is provided for external system integration to automatically create fault tickets.

[0063] The global dynamic topology graph, based on a force-directed layout algorithm, renders cluster node states, network connections, and data flow in real time. For example: Node color indicates status, such as green = normal, yellow = warning, and red = fault. Node size indicates load, with larger nodes indicating higher load. Edge thickness indicates traffic volume, with thicker edges indicating higher traffic volume. Edge color indicates traffic type, with blue for east-west and green for north-south.

[0064] It supports drilling down on nodes to view detailed resource information (such as CPU, memory, and disk usage) and a list of associated tasks.

[0065] Custom dashboards use drag-and-drop components, such as graphs, dashboards, and tables, to dynamically display resource utilization. For example: The graph displays CPU utilization trends over time and supports multi-dimensional filtering, such as by node, task, and time range. The dashboard displays key metrics, such as real-time values ​​for average cluster CPU utilization and memory utilization. The table displays task execution progress, such as the number of epochs completed and remaining time.

[0066] A task Gantt chart visually displays the dependencies and execution progress of tasks. For example: The Gantt chart shows that the "Data Loading" task starts at 10:00 and ends at 10:05; the "Model Training" task starts at 10:06 and is expected to end at 12:00.

[0067] It supports critical path analysis and bottleneck location. Critical path analysis can identify key tasks that affect the total duration of the task, while bottleneck location can identify tasks that take too long to load, causing overall progress delays.

[0068] Data traceability view: Based on task ID or file identifier, graphically traces the complete lifecycle and flow path of a task. For example: Enter the task ID "T123", and the system will display all the steps from task start to end, such as data loading → model training → parameter saving, and mark the time, status and path of each step.

[0069] It supports auditing and tracing, such as viewing the flow path of a file in a task, such as reading from HDFS → data preprocessing → inputting into the model layer → saving to local storage.

[0070] A method for visually monitoring the operational status and data flow of local large-scale model cluster resources includes: Step 1: Collect multi-dimensional runtime data from each node of the cluster by integrating hardware resource probes, application link tracers, job log collectors, and network traffic probes; Step 2: Use Kafka to build a high-throughput distributed message queue, and process data in parallel through multiple partitions to achieve traffic peak shaving and valley filling; Step 3: Implement real-time aggregation calculation and anomaly detection based on Flink, supporting dynamic configuration of threshold alarm rules; Step 4: Combine the stream processing results with historical data to trigger anomaly event alerts through the rule engine; Step 5: Use the force-directed layout algorithm to render the global dynamic topology map in real time, displaying node status, network connections and data flow.

[0071] Step 1 specifically includes: The hardware resource probe deploys Prometheus Node Exporter / cAdvisor to collect the utilization of hardware and container resources such as CPU, memory, disk, and network; the application link tracer injects Jaeger probes to automatically trace distributed task call chains and record module time, status, and path; the job log collector collects task logs and system logs through Fluentd; and the network traffic probe uses sFlow technology to capture east-west and north-south traffic to achieve full traffic capture.

[0072] Step 2 specifically includes: Hardware resource data is written to the "metrics-topic" partition, application link data is written to the "tracing-topic" partition, and log data is written to the "logs-topic" partition. The ISR mechanism ensures zero loss in real-time transmission of PB-level data, supports the decoupling of production and consumption in high-load scenarios, and achieves a peak throughput of 1GB / s.

[0073] Step 3 specifically includes: Consume Kafka data streams for real-time computation; store results in a multimodal database, and store time-series databases with timestamped resource metrics data; store logs and tracing data in Elasticsearch, supporting full-text search; and store dynamic relationships between nodes, services, and tasks in Neo4j graph database, supporting real-time topology graph generation.

[0074] Step 4 specifically includes: When the CPU utilization of a node exceeds 90% for 5 consecutive minutes, a P1-level alarm is triggered, and the operations team leader is notified through multiple channels such as WeChat, SMS, and email; alarm masking and confirmation mechanisms are supported; a unified API interface is provided to aggregate multi-source data for dynamic calling by front-end visualization components, avoiding direct access to multiple data sources by the front-end.

[0075] Step 5 specifically includes: Supports drilling down nodes to view resource details; enables dynamic display of resource utilization through custom dashboards with multi-dimensional filtering support; visualizes task dependencies and execution progress through task Gantt charts, supporting critical path analysis and bottleneck location; graphically traces the complete lifecycle and flow path based on task ID or file identifier through data traceability view; integrates alarm management functions, supports multi-channel notifications such as email / SMS / WeChat Work, and provides APIs for external system integration.

[0076] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A visualization monitoring system for the operational status and data flow of local large-scale model cluster resources, characterized in that, include: The data acquisition layer integrates hardware resource probes, application link tracers, job log collectors, and network traffic probes to collect multi-dimensional runtime data from various nodes in the cluster, which is used to capture all east-west and north-south traffic. The data transmission layer uses Kafka to build a high-throughput distributed message queue and processes the collected data in parallel through a partitioning mechanism. The data processing layer is based on Flink to achieve real-time aggregation computing and anomaly detection; it adopts a multimodal storage architecture, using a time-series database to store resource indicator data, Elasticsearch to store logs and tracing data, and Neo4j graph database to store the dynamic relationships between nodes, services, and tasks. The service layer combines stream processing results with historical data and uses a rule engine to enable real-time triggering of abnormal events; it also provides a unified interface for aggregating multi-source data. At the application layer, a force-directed layout algorithm is used to render a global dynamic topology map in real time, supporting node drilling to view resource details, and dynamically displaying resource utilization through a custom dashboard.

2. The visualization monitoring system for the operating status and data flow of local large-scale model cluster resources according to claim 1, characterized in that: In the data acquisition layer, the hardware resource probe collects CPU, memory, disk, network hardware and container resource utilization through Prometheus Node Exporter, the application link tracer tracks the distributed task call chain through Jaeger probe, the job log collector collects logs through Fluentd and stores them in Elasticsearch, and the network traffic probe captures inter-node traffic through sFlow and performs quintuple analysis.

3. The visualization monitoring system for the operating status and data flow of a local large-scale model cluster resources according to claim 2, characterized in that: The data acquisition layer employs a dynamic sampling rate adjustment mechanism, dynamically adjusting the sampling frequency based on the real-time load of the nodes. The calculation formula is as follows: ； Where S(t) is the sampling interval at the current time t, This is the upper limit of the sampling interval. Let L(t) be the lower limit of the sampling interval, and L(t) be the resource utilization rate of the processing unit at the current time t. σ is the load threshold, k is the adjustment coefficient, and σ is the smoothing factor.

4. The visualization monitoring system for the operating status and data flow of a local large-scale model cluster resources according to claim 3, characterized in that: The data acquisition layer performs edge computing preprocessing on the acquired raw data. By deploying lightweight Flink tasks on the node side, the acquired raw data is initially aggregated to reduce the amount of data transmitted to Kafka.

5. The visualization monitoring system for the operating status and data flow of a local large-scale model cluster resource according to claim 4, characterized in that: The number of Kafka partitions in the data transmission layer is automatically adjusted based on the consumption speed. When the consumer group's processing speed is lower than the production speed, Kafka automatically increases the number of partitions and triggers the Rebalance mechanism to reallocate partitions.

6. The visualization monitoring system for the operating status and data flow of a local large-scale model cluster resource as described in claim 5, characterized in that: In the storage layer, time-series data is partitioned by time and stored in InfluxDB hot data and S3 cold data. Data from the most recent 7 days is stored in InfluxDB, and data older than 7 days is archived in S3 and batch analyzed through Athena.

7. The visualization monitoring system for the operating status and data flow of a local large-scale model cluster resource according to claim 6, characterized in that: The Kafka producer side introduces a token bucket algorithm to limit burst traffic. It presets a maximum number of messages produced per second. When the traffic exceeds the preset maximum number of messages produced per second, it buffers and sends messages smoothly to avoid Broker overload.

8. A method for visually monitoring the operating status and data flow of local large-scale model cluster resources, used to implement the visual monitoring system for the operating status and data flow of local large-scale model cluster resources as described in any one of claims 1-7, characterized in that, include: Step 1: Collect multi-dimensional runtime data from each node of the cluster by integrating hardware resource probes, application link tracers, job log collectors, and network traffic probes; Step 2: Use Kafka to build a high-throughput distributed message queue, and process data in parallel through multiple partitions to achieve traffic peak shaving and valley filling; Step 3: Implement real-time aggregation calculation and anomaly detection based on Flink, supporting dynamic configuration of threshold alarm rules; Step 4: Combine the stream processing results with historical data to trigger anomaly event alerts through the rule engine; Step 5: Use the force-directed layout algorithm to render the global dynamic topology map in real time, displaying node status, network connections and data flow.

Images