A Multidimensional Real-Time Data Status Diagnosis and Analysis Method and System in a Cloud Environment

Abstract

The present invention relates to the technical field of data analysis, and discloses a multi-dimensional real-time data status diagnosis and analysis method and system in a cloud environment. The method includes: collecting multi-dimensional heterogeneous data, denoising, normalizing, imputing missing values and aligning timestamps; extracting features based on a windowed streaming computing framework to construct a matrix; using a graph attention network to construct a model to learn data associations and generate global representations; adopting a mixture density network and a variational autoencoder to construct an anomaly detection model, combining temporal causal inference to distinguish anomalies and generate scores; optimizing parameters based on a distributed policy gradient reinforcement learning algorithm; constructing a lightweight model through online knowledge distillation, and updating local model weights using federated learning. The invention can accurately process multi-dimensional real-time data, efficiently diagnose abnormal states, optimize system parameters in real time and protect data privacy, and improve the data processing and analysis capabilities in the cloud environment.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and particularly to a method and system for diagnosing and analyzing multi-dimensional real-time data status in a cloud environment. Background Art

[0002] With the wide application of cloud computing technology, a large amount of multi-dimensional real-time data has been generated in the cloud environment. These data come from different types of sensors, service instances, network devices, etc., and have the characteristics of large data volume, diverse types, and strong real-time performance. Effectively diagnosing and analyzing these multi-dimensional real-time data is crucial for ensuring the stable operation of cloud services, optimizing resource allocation, and improving user experience. However, current multi-dimensional real-time data processing in the cloud environment faces many challenges.

[0003] In terms of data collection and preprocessing, due to the heterogeneity of data sources, the formats and qualities of the collected data are uneven. There are differences in the sampling frequencies of different sensors, which makes it difficult to align the data in the time dimension and brings difficulties to subsequent unified analysis. At the same time, there are inevitably noises and missing values in the data. If not effectively processed, it will seriously affect the accuracy and reliability of data analysis. Traditional data denoising and missing value imputation methods often have poor effects when dealing with complex multi-dimensional data in the cloud environment and cannot fully utilize the spatio-temporal characteristics of the data.

[0004] There are also problems in the feature extraction link. The real-time data in the cloud environment has the characteristics of dynamic change. The traditional feature extraction methods with fixed windows and single aggregation operators are difficult to adapt to the rapid change of data and cannot comprehensively and timely capture the statistical features, frequency domain features, and time domain correlation features of the data. This results in the constructed feature vector matrix being unable to accurately reflect the internal laws of the data and affects subsequent data analysis and diagnosis.

[0005] For data status correlation analysis, the data relationships in the cloud environment are complex, involving various types of data sources and complex network topologies. Existing analysis methods are difficult to effectively mine the implicit associations between cross-dimensional data and cannot generate accurate global status representations, making the understanding and grasp of the overall state of the cloud environment not comprehensive and in-depth enough.

[0006] In terms of anomaly detection, due to the dynamics and uncertainties of the cloud environment, the manifestations of abnormal data are complex and diverse. Traditional anomaly detection models often can only be based on single indicators or simple pattern recognition methods, and are difficult to accurately identify complex abnormal patterns and cannot effectively distinguish instantaneous noises from persistent anomalies. In addition, existing anomaly detection models usually lack adaptability and cannot dynamically adjust the detection strategy according to the changes in the cloud environment, resulting in low accuracy and timeliness of detection.

[0007] In terms of system parameter optimization, the operating state of the cloud environment is constantly changing, and environmental factors such as CPU utilization, network latency, and data queue length are fluctuating all the time. The traditional fixed parameter setting method cannot adapt to this dynamic change, resulting in the system performance not reaching the optimal level. Moreover, in the cloud environment, a large number of computing tasks need to be coordinated between edge computing nodes and the cloud. How to achieve efficient parameter synchronization and model update while protecting data privacy is also an urgent problem to be solved. Summary of the Invention

[0008] The purpose of the present invention is to provide a multi-dimensional real-time data state diagnosis and analysis method and system in a cloud environment to solve the problems raised in the above background technology.

[0009] To achieve the above purpose, the present invention provides the following technical solutions: A multi-dimensional real-time data state diagnosis and analysis method in a cloud environment, the method includes:

[0010] Step 1: Real-time collect multi-dimensional heterogeneous data through a distributed sensor network, use an adaptive filtering algorithm to remove noise, and perform normalization and missing value imputation based on the data distribution characteristics; when the data sampling frequencies are inconsistent, use the dynamic time warping algorithm to align the timestamps to generate a standardized data stream with a unified time basis;

[0011] Step 2: Dynamically divide the real-time data stream based on a windowed streaming computing framework, and use a sliding time window combined with an incremental aggregation operator to extract statistical features, frequency domain features, and time domain correlation features to construct a multi-dimensional feature vector matrix;

[0012] Step 3: Use a graph attention network to construct a dynamic state association model, use the feature vector matrix as the node attribute, and the data source topological relationship as the edge weight, and learn the implicit association between cross-dimensional data through a multi-head attention mechanism to generate a global state representation;

[0013] Step 4: Use a mixture density network and a variational autoencoder to construct an anomaly detection model, identify data states deviating from the normal mode by jointly optimizing the reconstruction error and the distribution matching degree; combine a time series causal inference method to distinguish instantaneous noise from persistent anomalies and generate a multi-level anomaly confidence score;

[0014] Step 5: Based on the distributed policy gradient reinforcement learning algorithm, use the anomaly confidence score and the environmental state as inputs to dynamically adjust the feature extraction window size, model update frequency, and diagnostic threshold parameters to achieve real-time parameter adaptive optimization;

[0015] Step 6: Construct a lightweight diagnosis model through an online knowledge distillation mechanism, synchronize the global state representation and the optimized parameters to the edge computing nodes, and use a federated learning framework to update the local model weights to form a closed-loop feedback control link.

[0016] Preferably, the windowed streaming computing framework in step 2 is implemented using the Apache Flink engine. The length of the sliding time window is dynamically adjusted according to the data flow rate. The incremental aggregation operators include the exponentially weighted moving average, quantile estimation, and fast Fourier coefficient extraction algorithm.

[0017] Preferably, the graph attention network in step 3 adopts a heterogeneous subgraph partitioning strategy to construct subgraphs for sensor nodes, service instances, and network devices respectively, and fuses local and global state information through a cross-subgraph message passing mechanism to output an embedding vector with a dimension of 128.

[0018] Preferably, the mixture density network in step 4 is composed of a Gaussian mixture layer and a bidirectional long short-term memory network. The implicit space distribution constraint of the variational autoencoder is measured using the Wasserstein distance, and the temporal causal inference is realized through Granger causality test and Bayesian structure learning.

[0019] Preferably, the distributed policy gradient reinforcement learning algorithm in step 5 adopts the PPO framework. The policy network and the value network are respectively deployed on the cloud and edge nodes. The environmental state includes CPU utilization, network latency, and data queue length.

[0020] Preferably, the online knowledge distillation in step 6 adopts a teacher-student model collaborative training mechanism. The teacher model is the graph attention network in step 3, and the student model is a pruned lightweight convolutional neural network. The distillation loss function includes KL divergence and feature alignment loss terms.

[0021] Preferably, the missing value imputation in step 1 adopts a spatio-temporal Kriging interpolation algorithm. The spatial neighborhood weight is determined by a Gaussian kernel function, and the time decay factor follows an exponential distribution.

[0022] Preferably, the multi-level anomaly confidence score in step 4 includes device-level, service-level, and system-level scores. The device-level score is generated by the isolation forest algorithm. The service-level score propagates the anomaly impact weight based on the service dependency graph. The system-level score adopts the analytic hierarchy process to fuse multi-source indicators.

[0023] Preferably, the data source topological relationship in step 3 is dynamically constructed through service mesh link tracing data and Prometheus monitoring metrics. The edge weight update adopts a moving average strategy, and the decay coefficient is inversely proportional to the network traffic volatility.

[0024] Preferably, the present invention further includes a multi-dimensional real-time data state diagnosis and analysis system in a cloud environment. The system includes:

[0025] Data Acquisition and Preprocessing Module: Real-time collect multi-dimensional heterogeneous data through a distributed sensor network, use an adaptive filtering algorithm to remove noise, perform normalization and missing value imputation based on data distribution characteristics, and when the data sampling frequencies are inconsistent, use the dynamic time warping algorithm to align timestamps to generate a standardized data stream with a unified time benchmark;

[0026] Feature Extraction Module: Dynamically partition the real-time data stream based on a windowed streaming computing framework, and use a sliding time window combined with incremental aggregation operators to extract statistical features, frequency domain features, and time domain correlation features to construct a multi-dimensional feature vector matrix;

[0027] Dynamic State Association Modeling Module: Use a graph attention network to construct a dynamic state association model, use the feature vector matrix as node attributes and the data source topological relationship as edge weights, and learn the implicit associations between cross-dimensional data through a multi-head attention mechanism to generate a global state representation;

[0028] Anomaly Detection Module: Use a mixture density network and a variational autoencoder to construct an anomaly detection model, identify data states deviating from the normal pattern by jointly optimizing the reconstruction error and distribution matching degree, combine a time series causal inference method to distinguish instantaneous noise and persistent anomalies, and generate multi-level anomaly confidence scores;

[0029] Parameter Adaptive Optimization Module: Based on a distributed policy gradient reinforcement learning algorithm, use the anomaly confidence score and the environmental state as inputs to dynamically adjust the feature extraction window size, model update frequency, and diagnostic threshold parameters to achieve real-time parameter adaptive optimization;

[0030] Lightweight Diagnosis and Collaborative Update Module: Construct a lightweight diagnosis model through an online knowledge distillation mechanism, synchronize the global state representation and optimized parameters to edge computing nodes, and use a federated learning framework to update local model weights to form a closed-loop feedback control link.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] The present invention collects multi-dimensional heterogeneous data through a distributed sensor network, uses an adaptive filtering algorithm to remove noise, performs missing value imputation based on a spatio-temporal Kriging interpolation algorithm, and uses the dynamic time warping algorithm to align timestamps, and can generate a high-quality standardized data stream. This data preprocessing method fully considers the spatio-temporal characteristics of the data, effectively improves the data quality, and provides a reliable data basis for subsequent analysis. Compared with traditional methods, it can more accurately process complex multi-dimensional data and reduce the impact of data errors on the analysis results.

[0033] Based on the windowed streaming computing framework, by adopting a dynamically adjusted sliding time window combined with various incremental aggregation operators, such as exponentially weighted moving average, quantile estimation, and fast Fourier coefficient extraction algorithm, it can comprehensively and timely extract various features of data and construct a multi-dimensional feature vector matrix. This method can dynamically adjust the window length according to the data flow rate to adapt to the rapid changes of data. Compared with the traditional methods of fixed window and single aggregation operator, it can more accurately reflect the internal laws of data and provide richer and more representative feature information for subsequent data analysis.

[0034] Using a graph attention network to construct a dynamic state association model, adopting a heterogeneous subgraph partitioning strategy and a cross-subgraph message passing mechanism, combined with a multi-head attention mechanism, it can effectively learn the implicit associations between cross-dimensional data and generate accurate global state representations. This method can fully explore the complex data relationships in the cloud environment. Compared with traditional analysis methods, it can understand the overall state of the cloud environment more deeply and provide a more comprehensive basis for anomaly detection and system optimization.

[0035] Adopting a mixture density network and a variational autoencoder to construct an anomaly detection model, by jointly optimizing the reconstruction error and the distribution matching degree, combined with a time-series causal inference method of Granger causality test and Bayesian structure learning, it can accurately identify the data states deviating from the normal mode, effectively distinguish instantaneous noise from persistent anomalies, and generate multi-level anomaly confidence scores. Compared with traditional anomaly detection models, the method of the present invention can adapt to the dynamicity and uncertainty of the cloud environment, more accurately detect complex anomaly patterns, and improve the accuracy and reliability of anomaly detection.

[0036] Based on the distributed policy gradient reinforcement learning algorithm, taking the anomaly confidence score and the environmental state as inputs, it dynamically adjusts the feature extraction window size, model update frequency, and diagnostic threshold parameters. This real-time adaptive optimization method can timely adjust the system parameters according to the changes in the cloud environment, so that the system always maintains the optimal operating state. Compared with the traditional fixed parameter setting method, it greatly improves the performance and adaptability of the system.

[0037] By constructing a lightweight diagnostic model through an online knowledge distillation mechanism, adopting a teacher-student model collaborative training mechanism and a federated learning framework, synchronizing the global state representation and the optimized parameters to the edge computing nodes, and realizing the update of the local model weights. This method not only improves the diagnostic efficiency but also protects data privacy, meeting the dual requirements of data security and computing efficiency in the cloud environment. Compared with the traditional centralized computing and model update methods, it is more suitable for the application scenarios of large-scale cloud environments. Brief Description of the Drawings

[0038] Figure 1 It is the working principle diagram of the multi-dimensional real-time data state diagnosis and analysis method described in the present invention;

[0039] Figure 2 It is the workflow diagram of the windowed streaming computing framework;

[0040] Figure 3 It is the workflow diagram of the anomaly detection model

[0041] Figure 4 It is the workflow diagram of distributed policy gradient reinforcement learning. Specific implementation manners

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0043] Please refer to Figures 1-4 , the present invention provides a technical solution: a multi-dimensional real-time data status diagnosis and analysis method in a cloud environment, and the method includes:

[0044] Step 1: Real-time collect multi-dimensional heterogeneous data through a distributed sensor network, use an adaptive filtering algorithm to remove noise, and perform normalization and missing value imputation according to the data distribution characteristics. If the data sampling frequencies are inconsistent, align the timestamps with the help of the dynamic time warping algorithm to generate a standardized data stream with a unified time reference.

[0045] Step 2: Dynamically divide the real-time data stream based on the windowed streaming computing framework, and use a sliding time window combined with an incremental aggregation operator to extract statistical features, frequency domain features, and time domain correlation features, and construct a multi-dimensional feature vector matrix.

[0046] Step 3: Use a graph attention network to construct a dynamic state association model, use the feature vector matrix as the node attribute, and the data source topological relationship as the edge weight, and use the multi-head attention mechanism to learn the implicit association between cross-dimensional data, and then generate a global state representation.

[0047] Step 4: Use a mixture density network and a variational autoencoder to construct an anomaly detection model, identify data states deviating from the normal mode by jointly optimizing the reconstruction error and the distribution matching degree. Combine the time series causal inference method to distinguish instantaneous noise and persistent anomalies, and generate a multi-level anomaly confidence score.

[0048] Step 5: Based on the distributed policy gradient reinforcement learning algorithm, use the anomaly confidence score and the environmental state as inputs, and dynamically adjust the feature extraction window size, model update frequency, and diagnostic threshold parameters to achieve real-time parameter adaptive optimization.

[0049] Step 6: Build a lightweight diagnostic model through the online knowledge distillation mechanism, synchronize the global state representation and optimization parameters to the edge computing node, and use the federated learning framework to update the local model weights to form a closed-loop feedback control link.

[0050] The present invention will be further described below in conjunction with Examples 1 to 5: Example 1

[0051] In terms of data collection and preprocessing, distributed sensor networks are widely deployed in cloud environments to collect various types of heterogeneous data, such as physical quantity data collected by sensors, performance indicator data of servers, etc. Adaptive filtering algorithms automatically adjust filtering parameters based on real-time changes in data, effectively remove noise interference, and ensure data accuracy.

[0052] For missing value interpolation, the spatiotemporal kriging interpolation algorithm is used. In the spatial dimension, the Gaussian kernel function is used to determine the spatial neighborhood weight, and the formula is:

[0053]

[0054] in Indicates sensor and The spatial neighborhood weights between is the spatial distance between the two. is the bandwidth parameter of the Gaussian kernel function. The formula determines the weight according to the distance between sensors. The closer the distance, the greater the weight, so that the data of adjacent sensors can be used to interpolate missing values ​​more accurately. In the time dimension, the time decay factor follows an exponential distribution, which makes the recent data have a greater impact on the interpolation results, which is more in line with the dynamic change characteristics of the data.

[0055] When extracting features, the windowed streaming computing framework uses the Apache Flink engine. The length of the sliding time window is dynamically adjusted according to the data flow rate. When the data flow rate is fast, the window length is shortened to quickly capture short-term changes in the data; when the data flow rate is slow, the window length is extended to make full use of limited data. The exponentially weighted moving average (EWMA) in the incremental aggregation operator is calculated as follows:

[0056]

[0057] in is the weight coefficient, for The data value at the moment, for The exponentially weighted moving average of the time. , it can flexibly control the weight of recent data in the calculation and effectively reflect the trend changes of the data. The quantile estimation and the fast Fourier coefficient extraction algorithm work together to obtain the distribution characteristics and frequency domain characteristics of the data respectively, and jointly construct a multi-dimensional feature vector matrix to provide rich data features for subsequent analysis. Example 2

[0058] This embodiment elaborates in detail on the construction of the graph attention network and the processing of the topological relationship of the data source. The graph attention network adopts a heterogeneous subgraph partitioning strategy to construct subgraphs for sensor nodes, service instances, and network devices respectively.

[0059] For the sensor node subgraph, the data features collected by the sensors are used as node attributes, such as the measured values of sensors like temperature and pressure. The edge weights between nodes are determined according to the physical location of the sensors or the data correlation. For example, the edge weights between sensors with close geographical locations are larger, and the edge weights between sensors with similar data change trends also increase accordingly. In the service instance subgraph, the node attributes include information such as the running status and resource consumption of the service, such as CPU usage rate and memory occupancy. The edge weights are set based on the call relationship or dependency degree between services, and the edge weights between services that frequently call each other are large. The node attributes of the network device subgraph are the performance indicators of the network devices, such as bandwidth and latency, and the edge weights are determined according to the network topology structure and traffic relationship, and the edge weights of the links with larger traffic are higher.

[0060] Through the cross-subgraph message passing mechanism, information is exchanged between subgraphs to achieve the fusion of local and global state information. The graph attention network uses the multi-head attention mechanism to learn the implicit associations between cross-dimensional data and outputs an embedding vector with a dimension of 128 as the global state representation.

[0061] The topological relationship of the data source is dynamically constructed through the service mesh link tracing data and Prometheus monitoring metrics. The edge weight update adopts a moving average strategy, and the decay coefficient is inversely proportional to the network traffic volatility. Assume the decay coefficient is , and the network traffic volatility is , then ( is a constant). When the network traffic volatility is high, the decay coefficient is small, and the edge weight update is relatively slow to maintain the stability of the topological relationship; when the volatility is low, the decay coefficient is large, and the edge weights can reflect the changes in the network state in a timely manner. Example 3

[0062] The anomaly detection model consists of a mixture density network and a variational autoencoder. The mixture density network consists of a Gaussian mixture layer and a bidirectional long short-term memory network. The Gaussian mixture layer models the distribution of the data as a mixture of multiple Gaussian distributions, and the formula is:

[0063]

[0064] where is the number of Gaussian distributions, is the weight of the -th Gaussian distribution, is a Gaussian distribution with mean and covariance . By adjusting the parameters such as , , and , the complex distribution of the data can be accurately fitted. The bidirectional long short-term memory network is used to process the temporal information of the data and effectively capture the long-term dependence relationship of the data.

[0065] The implicit space distribution constraint of the variational autoencoder uses the Wasserstein distance metric, which can more accurately measure the difference between two probability distributions and has better mathematical properties and stability compared with other distance metric methods, helping to improve the performance of the model.

[0066] In distinguishing instantaneous noise from persistent anomalies, temporal causal inference is achieved through Granger causality test and Bayesian structure learning. The Granger causality test determines whether one time series has predictive power for another time series, thereby determining the causal relationship between the data. Bayesian structure learning constructs a Bayesian network to model the causal structure of the data and further analyze the causal relationship of the abnormal data.

[0067] The multi-level anomaly confidence score includes device-level, service-level, and system-level scores. The device-level score is generated by the isolation forest algorithm, which can quickly identify the abnormal points in the data. The service-level score propagates the abnormal impact weight based on the service dependency graph. According to the dependency relationship between services, the abnormal impact of a certain service is propagated to related services to evaluate the abnormal degree at the service level. The system-level score uses the analytic hierarchy process to fuse multi-source indicators, comprehensively considering the abnormal information at the device level and service level, as well as other system-level indicators, to comprehensively evaluate the anomaly confidence of the system. Example 4

[0068] The distributed policy gradient reinforcement learning algorithm adopts the PPO framework, with the policy network deployed in the cloud and the value network deployed at the edge nodes.

[0069] The policy network in the cloud receives anomaly confidence scores and environmental state information from multiple edge nodes. The environmental state includes CPU utilization, network latency, data queue length, etc. The policy network generates decision-making policies based on this information, such as adjusting the feature extraction window size, model update frequency, and diagnostic threshold parameters. For example, when the CPU utilization is too high, the policy network may decide to reduce the model update frequency to lower the CPU load; when the network latency is large, it decides to increase the feature extraction window size to reduce the number of data transmissions and relieve network pressure.

[0070] The value network of the edge node evaluates the value of the policy generated by the policy network based on the local environmental state. The PPO framework optimizes the policy network so that the policy network can dynamically adjust parameters according to the anomaly confidence score and environmental state. During the training process, the proximal policy optimization algorithm is adopted. This algorithm ensures the stability and convergence of the training process by restricting the amplitude of policy updates. By continuously interacting with the environment and learning, real-time adaptive optimization of system parameters is achieved, improving the performance of the system in different environments. Embodiment 5

[0071] This 5th embodiment details the collaborative update process of the online knowledge distillation mechanism and the lightweight diagnostic model. The online knowledge distillation adopts a teacher-student model collaborative training mechanism. The teacher model is a graph attention network, and the student model is a pruned lightweight convolutional neural network.

[0072] The teacher model has strong learning ability and can learn rich global state information. The student model learns knowledge from the teacher model through knowledge distillation. At the same time, due to its lightweight structure, it is suitable for running on edge computing nodes. The distillation loss function includes the KL divergence and the feature alignment loss term. The KL divergence is used to measure the difference between the output distributions of the student model and the teacher model. The formula is: , where is the output distribution of the teacher model, is the output distribution of the student model. By minimizing the KL divergence, the output of the student model is made as close as possible to that of the teacher model. The feature alignment loss term ensures the consistency between the student model and the teacher model at the feature level, further improving the performance of the student model.

[0073] During the collaborative update process, the global state representation and optimization parameters are synchronized to the edge computing nodes, and the local model weights are updated using the federated learning framework. The edge computing nodes utilize local data and the received global information to update the local model weights through the federated learning algorithm, and then upload the updated model parameters to the cloud. The cloud aggregates and optimizes the parameters uploaded by multiple edge computing nodes, and then distributes the optimized parameters to the edge computing nodes, forming a closed-loop feedback control link to continuously optimize the performance of the lightweight diagnostic model, while protecting data privacy and achieving efficient and secure multi-dimensional real-time data state diagnostic analysis in the cloud environment.

[0074] The present invention further includes a multi-dimensional real-time data state diagnostic analysis system in a cloud environment, and the system includes:

[0075] Data acquisition and preprocessing module: Real-time collect multi-dimensional heterogeneous data through a distributed sensor network, use an adaptive filtering algorithm to remove noise, perform normalization and missing value imputation based on data distribution characteristics, and when the data sampling frequencies are inconsistent, use the dynamic time warping algorithm to align time stamps to generate a standardized data stream with a unified time basis;

[0076] Feature extraction module: Dynamically divide the real-time data stream based on a windowed streaming computing framework, and use a sliding time window combined with an incremental aggregation operator to extract statistical features, frequency domain features, and time domain correlation features to construct a multi-dimensional feature vector matrix;

[0077] Dynamic state association modeling module: Use a graph attention network to construct a dynamic state association model, use the feature vector matrix as node attributes and the data source topological relationship as edge weights, and learn the implicit associations between cross-dimensional data through a multi-head attention mechanism to generate a global state representation;

[0078] Anomaly detection module: Use a mixture density network and a variational autoencoder to construct an anomaly detection model, identify data states deviating from the normal pattern by jointly optimizing the reconstruction error and distribution matching degree, and combine a time series causal inference method to distinguish instantaneous noise from persistent anomalies to generate multi-level anomaly confidence scores;

[0079] Parameter adaptive optimization module: Based on a distributed policy gradient reinforcement learning algorithm, use the anomaly confidence score and the environmental state as inputs to dynamically adjust the feature extraction window size, model update frequency, and diagnostic threshold parameters to achieve real-time parameter adaptive optimization;

[0080] Lightweight diagnosis and collaborative update module: Construct a lightweight diagnostic model through an online knowledge distillation mechanism, synchronize the global state representation and optimization parameters to the edge computing nodes, and use the federated learning framework to update the local model weights to form a closed-loop feedback control link.

[0081] The implementation of the system refers to the above-mentioned embodiments and will not be elaborated in the specification.

[0082] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0083] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-dimensional real-time data status diagnosis and analysis method in a cloud environment, characterized in that: The following steps are involved: Step 1: Collect multi-dimensional heterogeneous data in real time through a distributed sensor network, use an adaptive filtering algorithm to remove noise, and perform normalization and missing value interpolation based on data distribution characteristics; when the data sampling frequency is inconsistent, use a dynamic time warping algorithm to align timestamps and generate a standardized data stream with a unified time base; Step 2: Dynamically divide the real-time data stream into blocks based on the windowed streaming computing framework, use the sliding time window combined with the incremental aggregation operator to extract statistical features, frequency domain features, and time domain correlation features, and construct a multi-dimensional feature vector matrix; Step 3: Use the graph attention network to build a dynamic state association model, with the feature vector matrix as the node attribute and the data source topology relationship as the edge weight. The multi-head attention mechanism is used to learn the implicit association between cross-dimensional data and generate a global state representation. Step 4: Use a mixed density network and a variational autoencoder to build an anomaly detection model. By jointly optimizing the reconstruction error and distribution matching, the data state that deviates from the normal mode is identified. The time series causal inference method is combined to distinguish between instantaneous noise and persistent anomalies, and a multi-level anomaly confidence score is generated. Step 5: Based on the distributed policy gradient reinforcement learning algorithm, the anomaly confidence score and environmental state are used as input to dynamically adjust the feature extraction window size, model update frequency, and diagnostic threshold parameters to achieve real-time parameter adaptive optimization; Step 6: Build a lightweight diagnostic model through the online knowledge distillation mechanism, synchronize the global state representation and optimization parameters to the edge computing node, and use the federated learning framework to update the local model weights to form a closed-loop feedback control link.

2. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The windowed streaming computing framework described in step 2 is implemented using the Apache Flink engine. The length of the sliding time window is dynamically adjusted according to the data flow rate. The incremental aggregation operators include exponentially weighted moving average, quantile estimation, and Fourier coefficient fast extraction algorithm.

3. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The graph attention network described in step 3 adopts a heterogeneous subgraph partitioning strategy to construct subgraphs for sensor nodes, service instances, and network devices respectively, fuses local and global state information through a cross-subgraph message passing mechanism, and outputs an embedding vector with a dimension of 128.

4. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The mixed density network described in step 4 is composed of a Gaussian mixture layer and a bidirectional long short-term memory network. The latent space distribution constraint of the variational autoencoder adopts the Wasserstein distance metric, and the temporal causal inference is achieved through Granger causality test and Bayesian structure learning.

5. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The distributed policy gradient reinforcement learning algorithm described in step 5 adopts the PPO framework. The policy network and value network are deployed in the cloud and edge nodes respectively. The environmental status includes CPU utilization, network latency, and data queue length.

6. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The online knowledge distillation described in step 6 adopts a teacher-student model collaborative training mechanism. The teacher model is the graph attention network in step 3, and the student model is a pruned lightweight convolutional neural network. The distillation loss function includes KL divergence and feature alignment loss terms.

7. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The missing value interpolation described in step 1 adopts the spatiotemporal Kriging interpolation algorithm, the spatial neighborhood weight is determined by the Gaussian kernel function, and the time decay factor follows an exponential distribution.

8. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The multi-level anomaly confidence score described in step 4 includes device-level, service-level, and system-level scores. The device-level score is generated by the isolation forest algorithm, the service-level score propagates the anomaly impact weight based on the service dependency graph, and the system-level score uses the hierarchical analysis method to fuse multi-source indicators.

9. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The data source topology relationship described in step 3 is dynamically constructed through service mesh link tracking data and Prometheus monitoring indicators. The edge weight update adopts a sliding average strategy, and the attenuation coefficient is inversely proportional to the network traffic volatility.

10. A multi-dimensional real-time data status diagnosis and analysis system in a cloud environment, characterized in that: include: Data acquisition and preprocessing module: collects multi-dimensional heterogeneous data in real time through a distributed sensor network, uses an adaptive filtering algorithm to remove noise, performs normalization and missing value interpolation based on data distribution characteristics, and uses a dynamic time warping algorithm to align timestamps when data sampling frequencies are inconsistent, generating standardized data streams with a unified time base; Feature extraction module: Based on the windowed streaming computing framework, the real-time data stream is dynamically divided into blocks, and the sliding time window combined with the incremental aggregation operator is used to extract statistical features, frequency domain features, and time domain correlation features to construct a multi-dimensional feature vector matrix; Dynamic state association modeling module: Use graph attention network to build dynamic state association model, take feature vector matrix as node attribute and data source topology relationship as edge weight, learn implicit association between cross-dimensional data through multi-head attention mechanism, and generate global state representation; Anomaly detection module: A hybrid density network and variational autoencoder are used to build an anomaly detection model. By jointly optimizing the reconstruction error and distribution matching, the data state that deviates from the normal mode is identified. The time series causal inference method is combined to distinguish between instantaneous noise and persistent anomalies, and a multi-level anomaly confidence score is generated. Parameter adaptive optimization module: Based on the distributed policy gradient reinforcement learning algorithm, it takes the anomaly confidence score and environmental status as input, dynamically adjusts the feature extraction window size, model update frequency and diagnosis threshold parameters, and realizes real-time parameter adaptive optimization; Lightweight diagnosis and collaborative update module: A lightweight diagnosis model is constructed through an online knowledge distillation mechanism, the global state representation and optimization parameters are synchronized to the edge computing node, and the local model weights are updated using a federated learning framework to form a closed-loop feedback control link.

Images