Operation and maintenance data management system based on cloud computing

Abstract

The invention discloses an operation and maintenance data management system based on cloud computing, particularly relates to the field of data management, and comprises a distributed data acquisition module, a cloud native data processing module, an intelligent analysis module, a visualization and decision module and a cloud resource management center module. According to the method, heterogeneous data are collected in real time through the lightweight agent, and high efficiency is guaranteed by adopting multi-thread and self-adaptive sampling; data cleaning, standardization and mixed storage are realized by using a stream batch integrated architecture; the intelligent analysis module fuses an improved Robust Z-Score, an LSTM-Attention model and a complex event processing engine, and supports dynamic anomaly detection and fault tracing; the visual assembly builds an instrument panel in a dragging mode, alarm strategies are aggregated in a grading mode, and storm is restrained; the cloud resource management center deeply integrates Kubernetes, predicts demands based on reinforcement learning, automatically expands and shrinks capacity and optimizes cost, and deals with burst traffic in combination with a burst capacity pool.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and more particularly, to a cloud computing-based operation and maintenance data management system. BACKGROUND

[0002] With the deepening of digital transformation, enterprise IT architecture is rapidly evolving from traditional localized deployment to cloud-native and distributed architecture, and the complexity, scale and data volume of business systems are growing exponentially. Under this background, operation and maintenance data, as the core asset supporting system stable operation, fault troubleshooting and performance optimization, its management difficulty is increasing day by day, and the traditional operation and maintenance data management mode has been difficult to meet the needs of modern IT operation and maintenance.

[0003] The traditional operation and maintenance mode faces three core challenges: fragmented storage of data leads to lack of global perspective, manual processing efficiency is low and cannot cope with massive heterogeneous data, and lagging response cannot meet the real-time fault troubleshooting demand. In modern business scenarios, logs, performance indicators, configuration changes and other multi-source data generated by servers, containers, network devices and applications are growing explosively, and are scattered in different regions and platforms, forming "data islands". SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a cloud computing-based operation and maintenance data management system to solve the problems raised in the background art.

[0005] To achieve the above object, the present application provides the following technical scheme:

[0006] Distributed data acquisition module: deployed in the monitored physical devices, virtual machines, containers and cloud services, configured to collect heterogeneous operation and maintenance data sources in real time through a lightweight agent program;

[0007] Cloud-native data processing module: built on a cloud platform based on a microservice architecture, including:

[0008] Dynamic preprocessing unit: real-time cleaning, format standardization, key field extraction and data labeling of collected heterogeneous operation and maintenance data sources;

[0009] Elastic storage layer unit: adopts a hybrid storage model, including: object storage service for long-term archiving of raw logs; time series database cluster for high-performance storage and query of performance indicator data in operation and maintenance data sources; graph database for storing device topology relationship and dependency chain;

[0010] Intelligent analysis module: window aggregation and complex event detection on real-time data stream of heterogeneous operation and maintenance data sources; training of abnormal detection model based on historical data, dynamic identification of performance deviation and fault mode; correlation of multi-dimensional data, generation of topology-aware fault propagation path atlas;

[0011] Visualization and Decision Module: for customizing dynamic dashboard, combining prediction results with real-time thresholds to generate hierarchical alarms; API gateway opens analysis results to third-party operation and maintenance tools.

[0012] Preferably, in the distributed data collection module, the operation and maintenance data sources include log files, performance indicators, network traffic, configuration change events, and out-of-band management device data; a multi-thread architecture is adopted, and each collection thread is independently responsible for the capture task of a specific type of data, and a ring buffer is used for temporary storage to avoid data loss; the module has a built-in adaptive sampling mechanism that dynamically adjusts the collection frequency according to the resource usage of the monitored object - when the CPU utilization exceeds the preset threshold, it automatically switches to a reduced frequency sampling mode, while maintaining continuous collection of key indicators; for network traffic data, the module integrates deep packet inspection (DPI) capabilities to identify and extract application layer protocol features and perform correlation analysis with NetFlow data streams.

[0013] Preferably, in the cloud-native data processing module, in the dynamic preprocessing unit, a stream processing framework is used to build a data pipeline; in the data cleaning stage, the unit has an intelligent dirty data identification mechanism: a value range distribution model is established for each field through statistical learning methods to automatically detect and remove outliers beyond the 3σ range; for text log data, NLP technology is used to identify and correct common spelling errors and encoding chaos; the format standardization engine contains 200+ pre-built log format templates that automatically match heterogeneous logs to a unified JSON Schema structure, while preserving complete copies of heterogeneous operation and maintenance data sources for auditing; the key field extraction module uses a multi-modal processing strategy: structured data is directly mapped to fields; for semi-structured data (such as Nginx access logs), the FP-Growth algorithm-based frequent pattern mining is applied to automatically discover and extract key field combinations;

[0014] For unstructured data, a pre-trained deep learning model is used for semantic segment labeling; the data labeling system performs multi-dimensional metadata injection, including time dimension, space dimension, business dimension, etc., and supports integration with distributed tracing context through OpenTelemetry specifications.

[0015] Preferably, in the intelligent analysis module, a hierarchical processing architecture is adopted to integrate stream-batch integrated analysis capabilities. In the real-time analysis layer, the window aggregation engine implements an adaptive time window algorithm, and the window size is dynamically adjusted according to the data fluctuation rate:

[0016]

[0017] where W tW represents the current time window size. base Represented as the baseline window size, σ current σ represents the standard deviation of the current data. history This is expressed as the standard deviation of historical data;

[0018] Complex event detection employs a multi-pattern matching engine that supports nested definitions of CEP (Complex Event Processing) rules. For example, when detecting a complex event pattern of "CPU continuously exceeding the threshold and accompanied by disk I / O saturation", it uses a Petri net state machine to perform precise matching of event sequences.

[0019] In the anomaly detection model, short-term detection employs an improved Robust Z-Score algorithm, the specific calculation method of which is as follows:

[0020]

[0021] Among them, Z t Let x be the anomaly score at time t. t Let x represent the current data point at time t. w MAD is represented as the median value in the sliding window. w Expressed as the midpoint standard deviation within the window;

[0022] Long-term trend analysis uses an LSTM-ATTention combined model, with attention weights revealing key influencing dimensions. The specific calculation method is as follows:

[0023]

[0024] Where, α t Let W1 represent the attention weight at time t, and h represent the query weight matrix. t The hidden state at time t is represented as the hidden state output by the LSTM model after processing the t-th input data, which contains the sequence information of that time step and the preceding time steps; T represents the bias.

[0025] Preferably, the visualization and decision-making module adopts a responsive architecture design. The dynamic dashboard unit provides a drag-and-drop editor based on React, allowing users to freely combine more than 20 visualization components such as time series curves, topology maps, and Sankey diagrams. It also supports defining the linkage relationship between components through declarative syntax (such as "when a node in the topology map is selected, automatically filter the indicator curves associated with that node"). It has a built-in intelligent layout engine that automatically adjusts the dashboard layout according to the screen size and device type, and recommends the optimal visualization scheme for different data types such as KPI, logs, and topology.

[0026] The alarm decision engine employs a multi-level early warning mechanism: for predictive anomalies (such as an LSTM model predicting that the disk will be full in 1 hour), a "preliminary warning alarm" is triggered; for events that have exceeded the threshold but have not caused business impact, a "general alarm" is generated; for faults that cause service degradation, they are upgraded to "critical alarms"; the alarm aggregation unit applies a graph clustering algorithm to merge multiple alarms generated by the same fault root cause into an aggregated event, and outputs an alarm summary containing root cause inferences through natural language generation technology, such as "MySQL master-slave delay caused order service timeout (related to 5 alarms)"; the system implements an intelligent alarm suppression strategy, which automatically suppresses downstream application alarms when a data center-level fault is detected to avoid alarm storms.

[0027] Preferably, the cloud resource management hub module deeply integrates the Kubernetes orchestration system with the cloud platform API to build a full-stack resource awareness system. This module collects monitoring metrics of over 500 dimensions, such as CPU / memory / GPU utilization, network throughput, and storage IOPS at the container level in real time. It predicts resource demand inflection points 15 minutes in advance using a time series prediction model. The automatic scaling engine adopts a dual-threshold elastic strategy: when resource utilization exceeds 75% for 5 consecutive minutes, it triggers gradual scaling, while when the instantaneous peak exceeds 90%, it immediately doubles the number of instances. When a sudden traffic surge (such as a DDoS attack) is detected, it triggers emergency scaling.

[0028] For stateful services, the module implements intelligent drainage and graceful termination mechanisms to ensure no business data is lost during instance replacement. The failover subunit, through instance distribution across availability zones and health check probes, can complete service reconstruction within 30 seconds after a node failure is detected, and combines etcd's distributed locks to ensure topology consistency.

[0029] The health status detection uses heart rate monitoring combined with application-level probes for acquisition, and the calculation method is as follows:

[0030]

[0031] HealthScore represents the health score, HTTP 200 Represented as HTTP 200 Status codes represent the "success" response returned by the service after the application-level probe sends an HTTP request; Latency represents the response delay, which is the time interval between the probe sending the request and receiving the service response; ∧ represents the logical AND operator, meaning that both conditions must be met simultaneously; TCP connect This indicates a successful TCP connection, meaning the network layer probe successfully established a TCP connection with the service.

[0032] When the service returns an HTTP 200 (application is normal) and the response latency is less than 50ms, the health level is 1, which means it is completely healthy.

[0033] When a service can only establish TCP connections (network reachable but application may be partially abnormal) and the latency is <200ms, its health level is 0.5, which is considered sub-healthy.

[0034] In other cases (such as HTTP non-200, TCP connection failure, excessive latency, etc.), the health level is 0, which means the system is unhealthy at this moment.

[0035] In the fault recovery strategy, when a Pod-level failure occurs, an automatic restart is performed;

[0036] When encountering a node-level failure, it automatically migrates to a healthy node and optimizes resource allocation based on the Bin Packing algorithm. The specific calculation method is as follows:

[0037]

[0038] Where min represents the minimization function, and Q represents the total number of nodes, that is, the number of all healthy nodes participating in resource allocation; This represents the actual CPU utilization of node i. CPU resources allocated to node i;

[0039] The technical effects and advantages of this invention are as follows:

[0040] This invention achieves unified acquisition of multi-source heterogeneous data through a distributed acquisition module, and ensures data integrity using adaptive sampling and breakpoint resume technology. The cloud-native processing module includes a dynamic preprocessing unit and an elastic storage layer, employing a streaming processing framework and a hybrid storage model for efficient data processing. The intelligent analysis module integrates real-time stream computing and machine learning algorithms, achieving accurate fault prediction and root cause localization through improved PageRank and LSTM-ATTention models. The visualization module provides interactive dashboards and intelligent alarm classification, and the API gateway supports multi-tenant data access. The resource management hub uses reinforcement learning to achieve automatic scaling, and optimizes resource scheduling using health detection and Bin Packing algorithms. The system as a whole achieves intelligent operation and maintenance across the entire chain from data acquisition, processing, and analysis to decision-making, significantly improving fault response speed and resource utilization.

[0041] This invention achieves full-scale operation and maintenance data management through distributed collection and cloud-native processing, integrates adaptive algorithms and AI models to accurately detect anomalies, ensures high availability through dynamic scaling and intelligent fault transfer, improves decision-making efficiency through multi-level alarms and visualization, optimizes resource allocation to reduce costs through reinforcement learning, and supports flexible expansion and third-party integration. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the module connection of the present invention.

[0043] Figure 2 This is a schematic diagram of the intelligent analysis module of the present invention.

[0044] Figure 3 This is a schematic diagram of the visualization and decision-making alarm process of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figure 1 As shown, the present invention provides a cloud computing-based operation and maintenance data management system, including a distributed data acquisition module, a cloud-native data processing module, an intelligent analysis module, a visualization and decision-making module, and a cloud resource management hub module.

[0047] Distributed data acquisition module: Deployed in monitored physical devices, virtual machines, containers and cloud services, configured to collect heterogeneous operation and maintenance data sources in real time through a lightweight agent program;

[0048] In the distributed data acquisition module, the operation and maintenance data sources include log files, performance indicators, network traffic, configuration change events, and out-of-band management device data. It adopts a multi-threaded architecture, with each acquisition thread independently responsible for capturing specific types of data, and a circular buffer is used for temporary data storage to prevent data loss. The module has a built-in adaptive sampling mechanism that dynamically adjusts the acquisition frequency based on the resource usage of the monitored object—automatically switching to a reduced-frequency sampling mode when CPU utilization exceeds a preset threshold, while maintaining continuous acquisition of key indicators. For network traffic data, the module integrates deep packet inspection (DPI) capabilities to identify and extract application layer protocol features and perform correlation analysis with NetFlow data streams.

[0049] For out-of-band management device data acquisition, the module obtains hardware-level sensor data (such as fan speed and power supply voltage) through the IPMI / BMC interface and supplements it with device asset information via the Redfish API. All acquired data undergoes initial timestamp alignment and metadata tagging (including data source ID, acquisition period, and data confidence score) locally before being transmitted to the cloud processing layer via a compressed and encrypted channel. The module also implements a breakpoint resume function, automatically caching data to the local SSD when the network is interrupted and prioritizing the transmission of high-priority data after the connection is restored. In addition, the module provides a remote configuration hot update interface, allowing the cloud control center to dynamically adjust the acquisition strategy without restarting the agent program.

[0050] Cloud-native data processing module: Built on a cloud platform based on a microservice architecture, including:

[0051] Dynamic preprocessing unit: performs real-time cleaning, format standardization, key field extraction, and data tagging on the collected heterogeneous operation and maintenance data sources;

[0052] Elastic Storage Layer Unit: Employs a hybrid storage model, including: object storage service for long-term archiving of raw logs; time-series database cluster for high-performance storage and querying of performance metrics data from operational data sources; and graph database for storing device topology relationships and dependency chains.

[0053] In the cloud-native data processing module, the dynamic preprocessing unit employs a streaming processing framework for pipelined data processing. During the data cleaning phase, the unit incorporates an intelligent dirty data identification mechanism: it establishes a value range distribution model for each field using statistical learning methods, automatically detecting and removing outliers exceeding the 3σ range. For text-based log data, NLP technology is applied to identify and correct common spelling errors and encoding issues. The format standardization engine includes over 200 pre-defined log format templates, using automatic template matching technology to convert heterogeneous logs into a unified JSON Schema structure, while retaining a complete copy of the heterogeneous operation and maintenance data source for auditing. The key field extraction module adopts a multimodal processing strategy: it directly maps fields to structured data; for semi-structured data (such as Nginx access logs), it applies frequent pattern mining based on the FP-Growth algorithm to automatically discover and extract key field combinations.

[0054] Semantic fragment annotation is performed on unstructured data using pre-trained deep learning models; the data labeling system injects multi-dimensional metadata, including time, space, and business dimensions, and supports the integration of distributed tracking context through the OpenTelemetry specification;

[0055] In the elastic storage tier unit, the intelligent data routing engine coordinates the hybrid storage model. This engine automatically allocates the optimal storage location based on data type, access mode, and retention policy. The object storage service uses sharded compression storage technology, which divides the raw log into blocks according to time windows, compresses them using the Zstandard algorithm, and stores them while creating a min-max index for each block to accelerate range queries.

[0056] The time-series database cluster adopts a distributed architecture, with each node using columnar storage and time-sharded. For high-frequency query scenarios, the cluster has a built-in multi-layer caching mechanism: a full cache of the latest 5 minutes of data is maintained in memory, a hot time series is cached in an SSD layer, and a Bloom filter is used to quickly determine the query hit rate.

[0057] The graph database uses an attribute graph model to maintain the topology relationships between devices in real time; it maintains data consistency with the CMDB through a two-way synchronization mechanism, and automatically triggers topology reconstruction when a new or offline network device is detected; the storage layer also includes an intelligent lifecycle management module, which automatically migrates data according to predefined strategies: cools down indicator data older than 30 days from the time series database to object storage, and moves logs that have not been accessed for 6 months to archive storage.

[0058] Intelligent Analysis Module: Performs window aggregation and complex event detection on real-time data streams from heterogeneous operation and maintenance data sources; trains anomaly detection models based on historical data to dynamically identify performance deviations and failure modes; and correlates multi-dimensional data to generate a topology-aware fault propagation path map.

[0059] The intelligent analysis module employs a layered processing architecture, integrating stream and batch analysis capabilities. In the real-time analysis layer, a window aggregation engine implements an adaptive time window algorithm, dynamically adjusting the window size based on data volatility.

[0060]

[0061] Among them, W t W represents the current time window size. base Represented as the baseline window size, σ current σ represents the standard deviation of the current data. history This is expressed as the standard deviation of historical data;

[0062] Complex event detection employs a multi-pattern matching engine that supports nested definitions of CEP (Complex Event Processing) rules. For example, when detecting a complex event pattern of "CPU continuously exceeding the threshold and accompanied by disk I / O saturation", it uses a Petri net state machine to perform precise matching of event sequences.

[0063] In the anomaly detection model, short-term detection employs an improved Robust Z-Score algorithm, the specific calculation method of which is as follows:

[0064]

[0065] Among them, Z t Let x be the anomaly score at time t. t Let x represent the current data point at time t. w MAD is represented as the median value in the sliding window. w Expressed as the midpoint standard deviation within the window;

[0066] Long-term trend analysis uses an LSTM-ATTention combined model, with attention weights revealing key influencing dimensions. The specific calculation method is as follows:

[0067]

[0068] Where, α t Let W1 represent the attention weight at time t, and h represent the query weight matrix. t The hidden state at time t represents the hidden state output by the LSTM model after processing the t-th input data, which contains the sequence information of that time step and the preceding time steps; T represents the bias.

[0069] For cyclical indicators, the Facebook Prophet framework is integrated to decompose seasonal items;

[0070] Fault propagation analysis constructs a propagation graph based on an improved PageRank algorithm, and a time decay factor is introduced into the calculation of node importance scores:

[0071]

[0072] Where PR(u) represents the importance score of node u, d represents the damping coefficient, N represents the total number of nodes, and B u Let B be the set of backlink nodes pointing to node u, L(v) be the out-degree of node v, and λ be the number of other nodes directly pointed to by node v; λ be the attenuation coefficient, Δt be the fault interval, and v be the set B. u A node in the array represents a predecessor node that can propagate a fault to u.

[0073] The module maintains a dynamic cause-effect graph in real time. When a new fault is detected, it quickly generates Top-K suspected root cause paths through Monte Carlo tree search.

[0074] The analysis results are presented through three-dimensional visualization. The time dimension shows the evolution trend of indicators, the topology dimension highlights the fault propagation link, and the feature dimension shows the abnormal clustering after PCA dimensionality reduction. The system continuously monitors the model performance indicators and automatically triggers model retraining when concept drift is detected.

[0075] Visualization and Decision Module: Used to customize dynamic dashboards, generate tiered alarms by combining prediction results with real-time thresholds; API gateway opens analysis results to third-party operation and maintenance tools;

[0076] The visualization and decision-making module adopts a responsive architecture design. The dynamic dashboard unit provides a drag-and-drop editor based on React, allowing users to freely combine more than 20 visualization components such as time series curves, topology maps, and Sankey diagrams. It also supports defining the linkage relationship between components through declarative syntax (such as "when a node in the topology map is selected, automatically filter the indicator curves associated with that node"). The built-in intelligent layout engine automatically adjusts the dashboard layout according to the screen size and device type, and recommends the optimal visualization scheme for different data types such as KPIs, logs, and topology.

[0077] The alarm decision engine employs a multi-level early warning mechanism: for predictive anomalies (such as an LSTM model predicting that the disk will be full in 1 hour), a "preliminary warning alarm" is triggered; for events that have exceeded the threshold but have not caused business impact, a "general alarm" is generated; for faults that cause service degradation, they are upgraded to "critical alarms"; the alarm aggregation unit uses a graph clustering algorithm to merge multiple alarms generated by the same fault root cause into an aggregated event, and outputs an alarm summary containing root cause inferences through natural language generation technology, such as "MySQL master-slave delay caused order service timeout (related to 5 alarms)"; the system implements an intelligent alarm suppression strategy, which automatically suppresses downstream application alarms when a data center-level fault is detected to avoid alarm storms;

[0078] The API gateway adopts a GraphQL architecture, providing strongly typed data query interfaces and supporting third-party systems to obtain analysis results on demand. The gateway implements fine-grained access control, using JWT token authentication and OAuth 2.0 authorization mechanisms to grant differentiated data permissions to different tenants. For performance optimization, the gateway integrates the Apache Arrow data exchange format, improving the efficiency of large dataset transmission by 10 times, and uses a query caching mechanism to keep the response time of high-frequency queries within 50ms. To ensure system scalability, the gateway service adopts a service mesh architecture, allowing the number of instances to be dynamically increased or decreased based on load.

[0079] The cloud resource management hub module is used to dynamically monitor the system's cloud resource consumption, automatically scale up and down containerized components and perform failover, and then use reinforcement learning algorithms to predict resource requirements and pre-allocate computing instances.

[0080] The cloud resource management hub module deeply integrates the Kubernetes orchestration system with cloud platform APIs to build a full-stack resource awareness system. This module collects over 500 monitoring metrics in real time, including container-level CPU / memory / GPU utilization, network throughput, and storage IOPS. It uses a time-series prediction model to predict resource demand inflection points 15 minutes in advance. The automatic scaling engine employs a dual-threshold elasticity strategy: when resource utilization exceeds 75% for 5 consecutive minutes, it triggers gradual scaling; when the instantaneous peak exceeds 90%, it immediately doubles the number of instances. When a sudden traffic surge (such as a DDoS attack) is detected, it triggers emergency scaling. The specific calculation method is as follows:

[0081]

[0082] Among them, Replicas new Replicas represents the number of new replicas, min(·) represents the minimum value function. max Replicas represents the maximum number of replicas. current This represents the current number of replicas, i.e., the number of service instances running before the scaling operation; RequestRate represents the current request rate, and Threshold represents the request rate threshold.

[0083] For stateful services, the module implements intelligent drainage and graceful termination mechanisms to ensure no business data is lost during instance replacement. The failover subunit, through instance distribution across availability zones and health check probes, can complete service reconstruction within 30 seconds after a node failure is detected, and combines etcd's distributed locks to ensure topology consistency.

[0084] The health status detection uses heart rate monitoring combined with application-level probes for acquisition, and the calculation method is as follows:

[0085]

[0086] HealthScore represents the health score, HTTP 200 Represented as HTTP 200 Status codes represent the "success" response returned by the service after the application-level probe sends an HTTP request; Latency represents the response delay, which is the time interval between the probe sending the request and receiving the service response; ∧ represents the logical AND operator, meaning that both conditions must be met simultaneously; TCP connect This indicates a successful TCP connection, meaning the network layer probe successfully established a TCP connection with the service.

[0087] When the service returns an HTTP 200 (application is normal) and the response latency is less than 50ms, the health level is 1, which means it is completely healthy.

[0088] When a service can only establish TCP connections (network reachable but application may be partially abnormal) and the latency is <200ms, its health level is 0.5, which is considered sub-healthy.

[0089] In other cases (such as HTTP non-200, TCP connection failure, excessive latency, etc.), the health level is 0, which means the system is unhealthy at this moment.

[0090] In the fault recovery strategy, when a Pod-level failure occurs, an automatic restart is performed;

[0091] When encountering a node-level failure, it automatically migrates to a healthy node and optimizes resource allocation based on the Bin Packing algorithm. The specific calculation method is as follows:

[0092]

[0093] Where min represents the minimization function, and Q represents the total number of nodes, that is, the number of all healthy nodes participating in resource allocation; This represents the actual CPU utilization of node i. CPU resources allocated to node i;

[0094] In this formula, In this context, 'i' does not represent an exponent, but rather a label;

[0095] In the reinforcement learning predictor, historical load patterns, business calendar events, and real-time performance metrics serve as the state space, and a resource allocation strategy model is trained using the PPO algorithm. The module also includes a cost optimizer that intelligently mixes instances of different billing types based on Spot instance price fluctuations and reserved instance discount policies, reducing cloud resource costs by 30% while ensuring SLAs. All operations record change intentions through declarative APIs and are version-controlled and rolled back via GitOps workflows, while simultaneously synchronizing resource topology change information with the enterprise's CMDB system in real time.

[0096] To cope with sudden traffic surges, the module implements a "burst capacity pool" mechanism, which pre-maintains a pool of dormant preheated instances that can be quickly injected into the business cluster within 10 seconds. The resource scheduling decision-making process is simulated and displayed through a visual sandbox environment to help operations and maintenance personnel understand the automatic decision-making logic, and provides a manual intervention interface for strategy optimization in special scenarios.

[0097] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cloud computing-based operation and maintenance data management system, characterized in that, Comprise: Distributed data collection module: deployed in the monitored physical devices, virtual machines, containers and cloud services, configured to collect heterogeneous operation and maintenance data sources in real time through lightweight agent programs; Cloud native data processing module: based on microservices architecture built on cloud platform, including: Dynamic preprocessing unit: real-time cleaning, format standardization, key field extraction and data labeling of collected heterogeneous operation and maintenance data sources; Elastic storage layer unit: adopts hybrid storage model, including: object storage service for long-term archiving of raw logs; Time series database cluster for high-performance storage and query of performance indicator data in operation and maintenance data sources; Graph database for storing device topology relationship and dependency chain; Intelligent analysis module: window aggregation and complex event detection on real-time data stream of heterogeneous operation and maintenance data sources; based on historical data training anomaly detection model, dynamically identifying performance deviation and fault mode; correlate multi-dimensional data to generate topology-aware fault propagation path atlas; Visualization and decision module: for customizing dynamic dashboard, generating hierarchical alarms combined with prediction results and real-time thresholds; API gateway opens analysis results to third-party operation and maintenance tools; Cloud resource management hub module: for dynamically monitoring system cloud resource consumption, automatically scaling containers and failover, and predicting resource requirements and pre-allocating compute instances using reinforcement learning algorithm. 2.The cloud computing-based operation and maintenance data management system according to claim 1, characterized in that: In the distributed data collection module, operation and maintenance data sources include log files, performance indicators, network traffic, configuration change events, and out-of-band management device data; a multi-threaded architecture is adopted, with each collection thread independently responsible for a specific type of data capture task; the module has a built-in adaptive sampling mechanism that dynamically adjusts the collection frequency based on the resource usage of the monitored object - when the CPU utilization exceeds the preset threshold, it automatically switches to a reduced frequency sampling mode while maintaining continuous collection of key indicators; For network traffic data, the module integrates deep packet inspection capabilities to identify and extract application layer protocol features, and then performs correlation analysis with NetFlow data streams. 3.The cloud computing-based operation and maintenance data management system according to claim 1, characterized in that: In the cloud native data processing module, in the dynamic preprocessing unit, a stream processing framework is used to build a data pipeline; in the data cleaning stage, the unit has an intelligent dirty data identification mechanism: a value range distribution model is established for each field through statistical learning methods to automatically detect and remove outliers beyond the 3σ range; for text log data, NLP technology is used to identify and correct common spelling errors and encoding chaos; the format standardization engine includes 200+ pre-built log format templates that automatically match heterogeneous logs to a unified JSON Schema structure, while retaining complete copies of heterogeneous operation and maintenance data sources for auditing; The key field extraction module uses a multi-modal processing strategy: structured data is directly mapped to fields; for semi-structured data, the FP-Growth algorithm-based frequent pattern mining is applied to automatically discover and extract key field combinations.

4. The operation and maintenance data management system based on cloud computing according to claim 1, characterized in that: In the elastic storage layer unit, the mixed storage model works cooperatively through an intelligent data routing engine which automatically allocates the optimal storage location according to the data type, access mode and retention policy; the object storage service adopts a sharding compression storage technology, cuts the original log into blocks according to the time window, stores the compressed log using the Zstandard algorithm, and establishes a min-max index for each block to speed up the range query.

5. The operation and maintenance data management system based on cloud computing according to claim 1, characterized in that: In the intelligent analysis module, a hierarchical processing architecture is adopted to integrate the stream-batch integrated analysis capability; in the real-time analysis layer, a window aggregation engine implements an adaptive time window algorithm, and the window size is dynamically adjusted according to the data fluctuation rate; In the anomaly detection model, the short-term detection adopts an improved Robust Z-Score algorithm; The long-term trend analysis uses an LSTM-ATTention combined model, the attention weight reveals the key influence dimension, and the calculation method is as follows: wherein a t denotes the attention weight at time t, W1denotes the query weight matrix, h t denotes the hidden state at time t; The fault propagation analysis constructs a propagation graph based on an improved PageRank algorithm, and the node importance score calculation introduces a time decay factor; the module maintains a dynamic causal graph in real time, and when a new fault is detected, the Top-K suspicious root cause path is quickly generated through Monte Carlo tree search. 6.The cloud computing-based operation and maintenance data management system according to claim 1, characterized in that: In the visualization and decision module, the alarm decision engine implements a multi-level early warning mechanism: for predictive anomalies, a "pre-warning alarm" is triggered; For events that have exceeded the threshold but have not caused business impact, a "general alarm" is generated; For faults that cause service degradation, a "serious alarm" is upgraded; the alarm aggregation unit applies a graph clustering algorithm to combine multiple alarms generated by the same fault root into an aggregated event, and outputs an alarm summary containing root cause speculation through natural language generation technology, such as "MySQL master-slave delay causes order service timeout"; The system implements an alarm suppression intelligent strategy, which automatically suppresses downstream application alarms when a data center-level fault is detected to avoid alarm storms. 7.The cloud computing-based operation and maintenance data management system according to claim 1, characterized in that: In the cloud resource management hub module, real-time container-granularity monitoring indicators are collected, and a time series prediction model is used to predict resource demand inflection points 15 minutes in advance; the automatic scaling engine adopts a double-threshold elastic strategy: when the resource usage rate exceeds 75% for 5 minutes, gradual scaling is triggered, while an instantaneous peak exceeding 90% immediately doubles the number of instances; when a sudden traffic is detected, emergency scaling is triggered, and the calculation method is as follows: where Replicas new is the number of new replicas, min(·) is the minimum function, and Replicas max is the maximum number of replicas, and Replicas current is the current number of replicas, i.e., the number of instances the service was running before the scale-out operation; RequestRate is the current request rate, and Threshold is the request rate threshold. For stateful services, intelligent drainage and graceful termination mechanisms are implemented; the failover subunit completes service reconstruction within 30 seconds after detecting node failure through cross-availability zone instance distribution and health check probes; among them, the health degree detection uses heartbeat detection and application-level probes to obtain. 8.The cloud computing-based operation and maintenance data management system according to claim 7, characterized in that: In the fault recovery strategy, when a Pod-level fault is encountered, automatic restart is performed; When a node-level fault is encountered, automatic migration to a healthy node is performed, and resource allocation is optimized based on the Bin Packing algorithm; In the reinforcement learning predictor, the historical load pattern, business calendar events and real-time performance indicators are used as the state space, and the resource allocation strategy model is trained by the PPO algorithm; It also contains a cost optimizer that intelligently mixes instances of different billing types according to the Spot instance price fluctuations and reserved instance discount strategies. All operations record change intentions through declarative APIs and version control and rollback through GitOps workflows, while synchronizing resource topology change information with the enterprise CMDB system in real time.

Images