Multi-dimensional real-time data state diagnosis and analysis method and system in cloud environment

Abstract

The invention relates to the technical field of data analysis, and discloses a multi-dimensional real-time data state diagnostic analysis method and system in a cloud environment, and the method comprises the steps: collecting multi-dimensional heterogeneous data, denoising, normalizing, interpolating missing values, and aligning timestamps; extracting features based on a windowed streaming computing framework to construct a matrix; constructing a model by using a graph attention network to learn data association to generate global representation; adopting a mixed density network and a variational auto-encoder to construct an anomaly detection model, and combining time sequence causal inference to distinguish anomalies and generate scores; optimizing parameters based on a distributed strategy gradient reinforcement learning algorithm; a lightweight model is constructed through online knowledge distillation, and the weight of a local model is updated by adopting federal learning. According to the method, multi-dimensional real-time data can be accurately processed, an abnormal state can be efficiently diagnosed, system parameters are optimized in real time, data privacy is protected, and the data processing and analysis capability in a cloud environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a multi-dimensional real-time data status diagnosis and analysis method and system in a cloud environment. Background Art

[0002] With the widespread application of cloud computing technology, a massive amount of multi-dimensional real-time data is generated in the cloud environment. These data come from different types of sensors, service instances, and network devices, and have the characteristics of large data volume, diverse types, and strong real-time performance. Effective status diagnosis and analysis of these multi-dimensional real-time data are crucial to ensure the stable operation of cloud services, optimize resource allocation, and improve user experience. However, the current multi-dimensional real-time data processing in the cloud environment faces many challenges.

[0003] In terms of data collection and preprocessing, due to the heterogeneity of data sources, the format and quality of the collected data vary. The sampling frequencies of different sensors vary, which makes it difficult to align the data in the time dimension, bringing difficulties to subsequent unified analysis. At the same time, noise and missing values ​​are inevitably present in the data. If they are not effectively processed, the accuracy and reliability of data analysis will be seriously affected. Traditional data denoising and missing value interpolation methods often have poor effects when processing complex multidimensional data in cloud environments, and cannot fully utilize the spatiotemporal characteristics of the data.

[0004] There are also problems in the feature extraction process. Real-time data in cloud environments is characterized by dynamic changes. The traditional feature extraction method of fixed windows and single aggregation operators is difficult to adapt to the rapid changes of data and cannot fully and timely capture the statistical characteristics, frequency domain characteristics, and time domain correlation characteristics of the data. This results in the constructed feature vector matrix being unable to accurately reflect the inherent laws of the data, affecting subsequent data analysis and diagnosis.

[0005] For data state correlation analysis, the data relationships in the cloud environment are complex, involving multiple types of data sources and complex network topology structures. Existing analysis methods are difficult to effectively mine the implicit correlations between cross-dimensional data and cannot generate accurate global state representations, making the understanding and grasp of the overall state of the cloud environment not comprehensive and in-depth.

[0006] In terms of anomaly detection, due to the dynamics and uncertainty of the cloud environment, the manifestations of abnormal data are complex and diverse. Traditional anomaly detection models are often based on a single indicator or a simple pattern recognition method, which makes it difficult to accurately identify complex abnormal patterns and effectively distinguish between instantaneous noise and persistent anomalies. In addition, existing anomaly detection models usually lack adaptive capabilities and cannot dynamically adjust detection strategies according to changes in the cloud environment, resulting in low detection accuracy and timeliness.

[0007] In terms of system parameter optimization, the operating status of the cloud environment is constantly changing. Environmental factors such as CPU utilization, network latency, and data queue length are constantly fluctuating. The traditional fixed parameter setting method cannot adapt to this dynamic change, resulting in suboptimal system performance. Moreover, in the cloud environment, a large number of computing tasks need to be completed collaboratively between edge computing nodes and the cloud. How to achieve efficient parameter synchronization and model updates while protecting data privacy is also an urgent problem to be solved. Summary of the invention

[0008] The purpose of the present invention is to provide a multi-dimensional real-time data status diagnosis and analysis method and system in a cloud environment to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solution: a multi-dimensional real-time data status diagnosis and analysis method in a cloud environment, the method comprising: Step 1: Collect multi-dimensional heterogeneous data in real time through a distributed sensor network, use an adaptive filtering algorithm to remove noise, and perform normalization and missing value interpolation based on data distribution characteristics; when the data sampling frequency is inconsistent, use a dynamic time warping algorithm to align timestamps and generate a standardized data stream with a unified time base; Step 2: Dynamically divide the real-time data stream into blocks based on the windowed streaming computing framework, use the sliding time window combined with the incremental aggregation operator to extract statistical features, frequency domain features, and time domain correlation features, and construct a multi-dimensional feature vector matrix; Step 3: Use the graph attention network to build a dynamic state association model, with the feature vector matrix as the node attribute and the data source topology relationship as the edge weight. The multi-head attention mechanism is used to learn the implicit association between cross-dimensional data and generate a global state representation. Step 4: Use a mixed density network and a variational autoencoder to build an anomaly detection model. By jointly optimizing the reconstruction error and distribution matching, the data state that deviates from the normal mode is identified. The time series causal inference method is combined to distinguish between instantaneous noise and persistent anomalies, and a multi-level anomaly confidence score is generated. Step 5: Based on the distributed policy gradient reinforcement learning algorithm, the anomaly confidence score and environmental state are used as input to dynamically adjust the feature extraction window size, model update frequency, and diagnostic threshold parameters to achieve real-time parameter adaptive optimization; Step 6: Build a lightweight diagnostic model through the online knowledge distillation mechanism, synchronize the global state representation and optimization parameters to the edge computing node, and use the federated learning framework to update the local model weights to form a closed-loop feedback control link.

[0010] Preferably, the windowed streaming computing framework in step 2 is implemented using the Apache Flink engine, the length of the sliding time window is dynamically adjusted according to the data flow rate, and the incremental aggregation operators include exponentially weighted moving average, quantile estimation, and Fourier coefficient fast extraction algorithm.

[0011] Preferably, the graph attention network in step 3 adopts a heterogeneous subgraph partitioning strategy, constructs subgraphs for sensor nodes, service instances and network devices respectively, fuses local and global state information through a cross-subgraph message passing mechanism, and outputs an embedding vector with a dimension of 128.

[0012] Preferably, the mixed density network in step 4 is composed of a Gaussian mixing layer and a bidirectional long short-term memory network, the latent space distribution constraint of the variational autoencoder adopts the Wasserstein distance metric, and the temporal causal inference is achieved through Granger causality test and Bayesian structure learning.

[0013] Preferably, the distributed policy gradient reinforcement learning algorithm in step 5 adopts a PPO framework, the policy network and the value network are deployed in the cloud and edge nodes respectively, and the environmental status includes CPU utilization, network delay and data queue length.

[0014] Preferably, the online knowledge distillation described in step 6 adopts a teacher-student model collaborative training mechanism, the teacher model is the graph attention network in step 3, the student model is a pruned lightweight convolutional neural network, and the distillation loss function includes KL divergence and feature alignment loss terms.

[0015] Preferably, the missing value interpolation in step 1 adopts a spatiotemporal Kriging interpolation algorithm, the spatial neighborhood weight is determined by a Gaussian kernel function, and the time decay factor obeys an exponential distribution.

[0016] Preferably, the multi-level anomaly confidence score in step 4 includes device-level, service-level and system-level scores. The device-level score is generated by an isolation forest algorithm, the service-level score is based on a service dependency graph to propagate anomaly impact weights, and the system-level score uses a hierarchical analysis method to fuse multi-source indicators.

[0017] Preferably, the data source topology relationship in step 3 is dynamically constructed through service grid link tracking data and Prometheus monitoring indicators, and the edge weight update adopts a sliding average strategy, and the attenuation coefficient is inversely proportional to the network traffic volatility.

[0018] Preferably, the present invention also includes a multi-dimensional real-time data status diagnosis and analysis system in a cloud environment, the system comprising: Data acquisition and preprocessing module: collects multi-dimensional heterogeneous data in real time through a distributed sensor network, uses an adaptive filtering algorithm to remove noise, performs normalization and missing value interpolation based on data distribution characteristics, and uses a dynamic time warping algorithm to align timestamps when data sampling frequencies are inconsistent, generating standardized data streams with a unified time base; Feature extraction module: Based on the windowed streaming computing framework, the real-time data stream is dynamically divided into blocks, and the sliding time window combined with the incremental aggregation operator is used to extract statistical features, frequency domain features, and time domain correlation features to construct a multi-dimensional feature vector matrix; Dynamic state association modeling module: Use graph attention network to build dynamic state association model, take feature vector matrix as node attribute and data source topology relationship as edge weight, learn implicit association between cross-dimensional data through multi-head attention mechanism, and generate global state representation; Anomaly detection module: A hybrid density network and variational autoencoder are used to build an anomaly detection model. By jointly optimizing the reconstruction error and distribution matching, the data state that deviates from the normal mode is identified. The time series causal inference method is combined to distinguish between instantaneous noise and persistent anomalies, and a multi-level anomaly confidence score is generated. Parameter adaptive optimization module: Based on the distributed policy gradient reinforcement learning algorithm, it takes the anomaly confidence score and environmental status as input, dynamically adjusts the feature extraction window size, model update frequency and diagnosis threshold parameters, and realizes real-time parameter adaptive optimization; Lightweight diagnosis and collaborative update module: A lightweight diagnosis model is constructed through an online knowledge distillation mechanism, the global state representation and optimization parameters are synchronized to the edge computing node, and the local model weights are updated using a federated learning framework to form a closed-loop feedback control link.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention collects multi-dimensional heterogeneous data through a distributed sensor network, uses an adaptive filtering algorithm to remove noise, interpolates missing values ​​based on a spatiotemporal Kriging interpolation algorithm, and uses a dynamic time warping algorithm to align timestamps, thereby generating high-quality standardized data streams. This data preprocessing method fully considers the spatiotemporal characteristics of the data, effectively improves the data quality, and provides a reliable data basis for subsequent analysis. Compared with traditional methods, it can process complex multi-dimensional data more accurately and reduce the impact of data errors on analysis results.

[0020] Based on the windowed streaming computing framework, the dynamically adjusted sliding time window is combined with a variety of incremental aggregation operators, such as exponentially weighted moving average, quantile estimation and Fourier coefficient fast extraction algorithm, which can comprehensively and timely extract various data features and construct a multi-dimensional feature vector matrix. This method can dynamically adjust the window length according to the data flow rate to adapt to the rapid changes in data. Compared with the traditional fixed window and single aggregation operator method, it can more accurately reflect the inherent laws of the data and provide richer and more representative feature information for subsequent data analysis.

[0021] By using the graph attention network to build a dynamic state association model, adopting the heterogeneous subgraph partitioning strategy and the cross-subgraph message passing mechanism, combined with the multi-head attention mechanism, it can effectively learn the implicit associations between cross-dimensional data and generate accurate global state representations. This method can fully mine the complex data relationships in the cloud environment. Compared with traditional analysis methods, it can more deeply understand the overall state of the cloud environment and provide a more comprehensive basis for anomaly detection and system optimization.

[0022] The anomaly detection model is constructed by using a mixed density network and a variational autoencoder. By jointly optimizing the reconstruction error and distribution matching, and combining the Granger causality test with the temporal causal inference method of Bayesian structural learning, it can accurately identify data states that deviate from the normal mode, effectively distinguish between instantaneous noise and persistent anomalies, and generate multi-level anomaly confidence scores. Compared with traditional anomaly detection models, the method of the present invention can adapt to the dynamics and uncertainty of the cloud environment, more accurately detect complex anomaly patterns, and improve the accuracy and reliability of anomaly detection.

[0023] Based on the distributed policy gradient reinforcement learning algorithm, the feature extraction window size, model update frequency and diagnostic threshold parameters are dynamically adjusted with the abnormal confidence score and environmental status as input. This real-time adaptive optimization method can adjust system parameters in time according to changes in the cloud environment, so that the system always maintains the optimal operating state. Compared with the traditional fixed parameter setting method, it greatly improves the performance and adaptability of the system.

[0024] A lightweight diagnostic model is constructed through an online knowledge distillation mechanism. The teacher-student model collaborative training mechanism and federated learning framework are used to synchronize the global state representation and optimization parameters to the edge computing node to update the local model weights. This method not only improves diagnostic efficiency, but also protects data privacy, meeting the dual requirements of data security and computing efficiency in the cloud environment. Compared with traditional centralized computing and model updating methods, it is more suitable for application scenarios in large-scale cloud environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a working principle diagram of the multi-dimensional real-time data status diagnosis and analysis method of the present invention; Figure 2 This is a workflow diagram of the windowed streaming computing framework; Figure 3 Workflow diagram for anomaly detection model Figure 4 Workflow diagram for distributed policy gradient reinforcement learning. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] See also Figure 1-4 The present invention provides a technical solution: a multi-dimensional real-time data status diagnosis and analysis method under a cloud environment, the method comprising: Step 1: Collect multi-dimensional heterogeneous data in real time through a distributed sensor network, use an adaptive filtering algorithm to remove noise, and perform normalization and missing value interpolation based on data distribution characteristics. If the data sampling frequency is inconsistent, align the timestamps with the help of a dynamic time warping algorithm to generate a standardized data stream with a unified time reference.

[0028] Step 2: Dynamically divide the real-time data stream into blocks based on the windowed streaming computing framework, use the sliding time window combined with the incremental aggregation operator to extract statistical features, frequency domain features, and time domain correlation features, and construct a multi-dimensional feature vector matrix.

[0029] Step 3: Use the graph attention network to build a dynamic state association model, with the feature vector matrix as the node attribute and the data source topology relationship as the edge weight. Use the multi-head attention mechanism to learn the implicit association between cross-dimensional data and generate a global state representation.

[0030] Step 4: Use a mixed density network and a variational autoencoder to build an anomaly detection model, and identify data states that deviate from normal patterns by jointly optimizing reconstruction error and distribution matching. Combined with the time series causal inference method, it distinguishes between instantaneous noise and persistent anomalies and generates a multi-level anomaly confidence score.

[0031] Step 5: Based on the distributed policy gradient reinforcement learning algorithm, with the anomaly confidence score and environmental status as input, dynamically adjust the feature extraction window size, model update frequency and diagnosis threshold parameters to achieve real-time parameter adaptive optimization.

[0032] Step 6: Build a lightweight diagnostic model through the online knowledge distillation mechanism, synchronize the global state representation and optimization parameters to the edge computing node, and use the federated learning framework to update the local model weights to form a closed-loop feedback control link.

[0033] The present invention will be further described below in conjunction with Examples 1 to 5: Example 1

[0034] In terms of data collection and preprocessing, distributed sensor networks are widely deployed in cloud environments to collect various types of heterogeneous data, such as physical quantity data collected by sensors, performance indicator data of servers, etc. Adaptive filtering algorithms automatically adjust filtering parameters based on real-time changes in data, effectively remove noise interference, and ensure data accuracy.

[0035] For missing value interpolation, the spatiotemporal kriging interpolation algorithm is used. In the spatial dimension, the Gaussian kernel function is used to determine the spatial neighborhood weight, and the formula is:

[0036] in Indicates sensor and The spatial neighborhood weights between is the spatial distance between the two. is the bandwidth parameter of the Gaussian kernel function. The formula determines the weight according to the distance between sensors. The closer the distance, the greater the weight, so that the data of adjacent sensors can be used to interpolate missing values ​​more accurately. In the time dimension, the time decay factor follows an exponential distribution, which makes the recent data have a greater impact on the interpolation results, which is more in line with the dynamic change characteristics of the data.

[0037] When extracting features, the windowed streaming computing framework uses the Apache Flink engine. The length of the sliding time window is dynamically adjusted according to the data flow rate. When the data flow rate is fast, the window length is shortened to quickly capture short-term changes in the data; when the data flow rate is slow, the window length is extended to make full use of limited data. The exponentially weighted moving average (EWMA) in the incremental aggregation operator is calculated as follows:

[0038] in is the weight coefficient, for The data value at the moment, for The exponentially weighted moving average of the time. , can flexibly control the weight of recent data in the calculation, and effectively reflect the trend change of data. Quantile estimation and Fourier coefficient fast extraction algorithms work together to obtain the distribution characteristics and frequency domain characteristics of data, respectively, and jointly construct a multi-dimensional feature vector matrix to provide rich data features for subsequent analysis. Example 2

[0039] This embodiment elaborates on the construction of the graph attention network and the processing of the topological relationship of the data source. The graph attention network adopts a heterogeneous subgraph partitioning strategy to construct subgraphs for sensor nodes, service instances and network devices respectively.

[0040] For the sensor node subgraph, the data features collected by the sensor are used as node attributes, such as sensor measurement values ​​such as temperature and pressure. The edge weights between nodes are determined based on the physical location of the sensor or the data correlation. For example, the edge weights between sensors with similar geographical locations are larger, and the edge weights between sensors with similar data change trends are also increased accordingly. In the service instance subgraph, node attributes include information such as the running status of the service, resource consumption, such as CPU usage, memory usage, etc. The edge weights are set based on the calling relationship or degree of dependence between services. The edge weights between services that frequently call each other are large. The node attributes of the network device subgraph are the performance indicators of the network equipment, such as bandwidth, latency, etc. The edge weights are determined based on the network topology and traffic relationship. The edge weights of links with large traffic are higher.

[0041] Through the cross-subgraph message passing mechanism, subgraphs exchange information with each other to achieve the fusion of local and global state information. The graph attention network uses a multi-head attention mechanism to learn the implicit associations between cross-dimensional data and outputs an embedding vector with a dimension of 128 as the global state representation.

[0042] The data source topology is dynamically constructed through the service grid link tracking data and Prometheus monitoring indicators. The edge weight update adopts a sliding average strategy, and the attenuation coefficient is inversely proportional to the network traffic volatility. Assume that the attenuation coefficient is , the network traffic fluctuation rate is ,but ( is a constant). When the network traffic volatility is high, the attenuation coefficient is small, and the edge weight is updated relatively slowly to maintain the stability of the topological relationship; when the volatility is low, the attenuation coefficient is large, and the edge weight can reflect the changes in the network status in a timely manner. Example 3

[0043] The anomaly detection model consists of a mixture density network and a variational autoencoder. The mixture density network consists of a Gaussian mixture layer and a bidirectional long short-term memory network. The Gaussian mixture layer models the distribution of data as a mixture of multiple Gaussian distributions, and the formula is:

[0044] in is the number of Gaussian distributions, For the The weights of a Gaussian distribution, The mean is , the covariance is Gaussian distribution. By adjusting , , and Parameters such as , can accurately fit the complex distribution of data. The bidirectional long short-term memory network is used to process the temporal information of data and effectively capture the long-term dependencies of data.

[0045] The latent space distribution constraint of the variational autoencoder uses the Wasserstein distance metric, which can more accurately measure the difference between two probability distributions. Compared with other distance measurement methods, it has better mathematical properties and stability, which helps to improve the performance of the model.

[0046] In terms of distinguishing instantaneous noise from persistent anomalies, time series causal inference is achieved through Granger causality test and Bayesian structure learning. Granger causality test determines whether one time series has predictive power for another time series, thereby determining the causal relationship between data. Bayesian structure learning constructs a Bayesian network to model the causal structure of the data and further analyze the causal relationship of abnormal data.

[0047] The multi-level anomaly confidence score includes device-level, service-level, and system-level scores. The device-level score is generated by the Isolation Forest algorithm, which can quickly identify anomalies in the data. The service-level score propagates the anomaly impact weight based on the service dependency graph. According to the dependency relationship between services, the anomaly impact of a service is propagated to related services to evaluate the degree of anomaly at the service level. The system-level score uses the hierarchical analysis method to integrate multi-source indicators, comprehensively consider the anomaly information at the device level and service level, and other system-level indicators, and comprehensively evaluate the anomaly confidence of the system. Example 4

[0048] The distributed policy gradient reinforcement learning algorithm adopts the PPO framework, with the policy network deployed in the cloud and the value network deployed in the edge nodes.

[0049] The policy network in the cloud receives anomaly confidence scores and environmental status information from multiple edge nodes, including CPU utilization, network latency, and data queue length. The policy network generates decision strategies based on this information, such as adjusting the feature extraction window size, model update frequency, and diagnostic threshold parameters. For example, when the CPU utilization is too high, the policy network may decide to reduce the model update frequency to reduce the CPU load; when the network latency is large, the decision may be to increase the feature extraction window size, reduce the number of data transmissions, and relieve network pressure.

[0050] The value network of the edge node evaluates the value of the policy generated by the policy network according to the local environment state. The PPO framework optimizes the policy network so that the policy network can dynamically adjust parameters according to the abnormal confidence score and the environment state. During the training process, the proximal policy optimization algorithm is used, which ensures the stability and convergence of the training process by limiting the amplitude of the policy update. By continuously interacting and learning with the environment, the real-time adaptive optimization of system parameters is achieved, and the performance of the system in different environments is improved. Example 5

[0051] This fifth embodiment describes in detail the online knowledge distillation mechanism and the collaborative update process of the lightweight diagnosis model. Online knowledge distillation adopts a teacher-student model collaborative training mechanism, where the teacher model is a graph attention network and the student model is a pruned lightweight convolutional neural network.

[0052] The teacher model has a strong learning ability and can learn rich global state information. The student model learns knowledge from the teacher model through knowledge distillation. At the same time, due to its lightweight structure, it is suitable for running on edge computing nodes. The distillation loss function contains KL divergence and feature alignment loss terms. KL divergence is used to measure the difference between the output distribution of the student model and the teacher model. The formula is: ,in is the output distribution of the teacher model, is the output distribution of the student model. By minimizing the KL divergence, the output of the student model is made as close as possible to the teacher model. The feature alignment loss term ensures the consistency of the student model and the teacher model at the feature level, further improving the performance of the student model.

[0053] In the collaborative update process, the global state representation and optimization parameters are synchronized to the edge computing node, and the federated learning framework is used to update the local model weights. The edge computing node uses local data and the received global information to update the local model weights through the federated learning algorithm, and then uploads the updated model parameters to the cloud. The cloud aggregates and optimizes the parameters uploaded by multiple edge computing nodes, and then sends the optimized parameters to the edge computing nodes, forming a closed-loop feedback control link, continuously optimizing the performance of the lightweight diagnostic model, while protecting data privacy, and realizing efficient and secure multi-dimensional real-time data status diagnostic analysis in the cloud environment.

[0054] The present invention also includes a multi-dimensional real-time data status diagnosis and analysis system in a cloud environment, the system comprising: Data acquisition and preprocessing module: collects multi-dimensional heterogeneous data in real time through a distributed sensor network, uses an adaptive filtering algorithm to remove noise, performs normalization and missing value interpolation based on data distribution characteristics, and uses a dynamic time warping algorithm to align timestamps when data sampling frequencies are inconsistent, generating standardized data streams with a unified time base; Feature extraction module: Based on the windowed streaming computing framework, the real-time data stream is dynamically divided into blocks, and the sliding time window combined with the incremental aggregation operator is used to extract statistical features, frequency domain features, and time domain correlation features to construct a multi-dimensional feature vector matrix; Dynamic state association modeling module: Use graph attention network to build dynamic state association model, take feature vector matrix as node attribute and data source topology relationship as edge weight, learn implicit association between cross-dimensional data through multi-head attention mechanism, and generate global state representation; Anomaly detection module: A hybrid density network and variational autoencoder are used to build an anomaly detection model. By jointly optimizing the reconstruction error and distribution matching, the data state that deviates from the normal mode is identified. The time series causal inference method is combined to distinguish between instantaneous noise and persistent anomalies, and a multi-level anomaly confidence score is generated. Parameter adaptive optimization module: Based on the distributed policy gradient reinforcement learning algorithm, it takes the anomaly confidence score and environmental status as input, dynamically adjusts the feature extraction window size, model update frequency and diagnosis threshold parameters, and realizes real-time parameter adaptive optimization; Lightweight diagnosis and collaborative update module: A lightweight diagnosis model is constructed through an online knowledge distillation mechanism, the global state representation and optimization parameters are synchronized to the edge computing node, and the local model weights are updated using a federated learning framework to form a closed-loop feedback control link.

[0055] The implementation of the system refers to the above embodiment and will not be described in detail in the specification.

[0056] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-dimensional real-time data status diagnosis and analysis method in a cloud environment, characterized in that: The following steps are involved: Step 1: Collect multi-dimensional heterogeneous data in real time through a distributed sensor network, use an adaptive filtering algorithm to remove noise, and perform normalization and missing value interpolation based on data distribution characteristics; when the data sampling frequency is inconsistent, use a dynamic time warping algorithm to align timestamps and generate a standardized data stream with a unified time base; Step 2: Dynamically divide the real-time data stream into blocks based on the windowed streaming computing framework, use the sliding time window combined with the incremental aggregation operator to extract statistical features, frequency domain features, and time domain correlation features, and construct a multi-dimensional feature vector matrix; Step 3: Use the graph attention network to build a dynamic state association model, with the feature vector matrix as the node attribute and the data source topology relationship as the edge weight. The multi-head attention mechanism is used to learn the implicit association between cross-dimensional data and generate a global state representation. Step 4: Use a mixed density network and a variational autoencoder to build an anomaly detection model. By jointly optimizing the reconstruction error and distribution matching, the data state that deviates from the normal mode is identified. The time series causal inference method is combined to distinguish between instantaneous noise and persistent anomalies, and a multi-level anomaly confidence score is generated. Step 5: Based on the distributed policy gradient reinforcement learning algorithm, the anomaly confidence score and environmental state are used as input to dynamically adjust the feature extraction window size, model update frequency, and diagnostic threshold parameters to achieve real-time parameter adaptive optimization; Step 6: Build a lightweight diagnostic model through the online knowledge distillation mechanism, synchronize the global state representation and optimization parameters to the edge computing node, and use the federated learning framework to update the local model weights to form a closed-loop feedback control link.

2. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The windowed streaming computing framework described in step 2 is implemented using the Apache Flink engine. The length of the sliding time window is dynamically adjusted according to the data flow rate. The incremental aggregation operators include exponentially weighted moving average, quantile estimation, and Fourier coefficient fast extraction algorithm.

3. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The graph attention network described in step 3 adopts a heterogeneous subgraph partitioning strategy to construct subgraphs for sensor nodes, service instances, and network devices respectively, fuses local and global state information through a cross-subgraph message passing mechanism, and outputs an embedding vector with a dimension of 128.

4. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The mixed density network described in step 4 is composed of a Gaussian mixture layer and a bidirectional long short-term memory network. The latent space distribution constraint of the variational autoencoder adopts the Wasserstein distance metric, and the temporal causal inference is achieved through Granger causality test and Bayesian structure learning.

5. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The distributed policy gradient reinforcement learning algorithm described in step 5 adopts the PPO framework. The policy network and value network are deployed in the cloud and edge nodes respectively. The environmental status includes CPU utilization, network latency, and data queue length.

6. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The online knowledge distillation described in step 6 adopts a teacher-student model collaborative training mechanism. The teacher model is the graph attention network in step 3, and the student model is a pruned lightweight convolutional neural network. The distillation loss function includes KL divergence and feature alignment loss terms.

7. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The missing value interpolation described in step 1 adopts the spatiotemporal Kriging interpolation algorithm, the spatial neighborhood weight is determined by the Gaussian kernel function, and the time decay factor follows an exponential distribution.

8. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The multi-level anomaly confidence score described in step 4 includes device-level, service-level, and system-level scores. The device-level score is generated by the isolation forest algorithm, the service-level score propagates the anomaly impact weight based on the service dependency graph, and the system-level score uses the hierarchical analysis method to fuse multi-source indicators.

9. The multi-dimensional real-time data status diagnosis and analysis method under cloud environment according to claim 1, characterized in that: The data source topology relationship described in step 3 is dynamically constructed through service mesh link tracking data and Prometheus monitoring indicators. The edge weight update adopts a sliding average strategy, and the attenuation coefficient is inversely proportional to the network traffic volatility.

10. A multi-dimensional real-time data status diagnosis and analysis system in a cloud environment, characterized in that: include: Data acquisition and preprocessing module: collects multi-dimensional heterogeneous data in real time through a distributed sensor network, uses an adaptive filtering algorithm to remove noise, performs normalization and missing value interpolation based on data distribution characteristics, and uses a dynamic time warping algorithm to align timestamps when data sampling frequencies are inconsistent, generating standardized data streams with a unified time base; Feature extraction module: Based on the windowed streaming computing framework, the real-time data stream is dynamically divided into blocks, and the sliding time window combined with the incremental aggregation operator is used to extract statistical features, frequency domain features, and time domain correlation features to construct a multi-dimensional feature vector matrix; Dynamic state association modeling module: Use graph attention network to build dynamic state association model, take feature vector matrix as node attribute and data source topology relationship as edge weight, learn implicit association between cross-dimensional data through multi-head attention mechanism, and generate global state representation; Anomaly detection module: A hybrid density network and variational autoencoder are used to build an anomaly detection model. By jointly optimizing the reconstruction error and distribution matching, the data state that deviates from the normal mode is identified. The time series causal inference method is combined to distinguish between instantaneous noise and persistent anomalies, and a multi-level anomaly confidence score is generated. Parameter adaptive optimization module: Based on the distributed policy gradient reinforcement learning algorithm, it takes the anomaly confidence score and environmental status as input, dynamically adjusts the feature extraction window size, model update frequency and diagnosis threshold parameters, and realizes real-time parameter adaptive optimization; Lightweight diagnosis and collaborative update module: A lightweight diagnosis model is constructed through an online knowledge distillation mechanism, the global state representation and optimization parameters are synchronized to the edge computing node, and the local model weights are updated using a federated learning framework to form a closed-loop feedback control link.

Images