Cloud computing environment anomaly detection, storage, calculation and training integrated method based on variational auto-encoder

Abstract

The invention discloses a cloud computing environment anomaly detection, storage, calculation and training integrated method based on a variational auto-encoder, and the method comprises the steps: S1, designing and initializing a TensorFlow-based distributed variational auto-encoder model, and forming reconstruction data; s2, in a cloud computing environment, constructing a distributed data storage and computing architecture; s3, deploying and configuring a distributed computing framework of TensorFlow on the plurality of computing nodes, and configuring a parameter server and a worker computing node; s4, each computing node obtains a locally stored preprocessing data set in parallel by using a TensorFlow distributed strategy; s5, after training is completed, reconstruction data are generated through a decoder; and S6, comparing the detected reconstruction error with a preset threshold value by each computing node, and marking the data exceeding the threshold value as abnormal data. Through the integrated design, large-scale distributed data anomaly detection in the cloud computing environment is successfully realized.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a storage, computing and training integrated method for anomaly detection in a cloud computing environment based on a variational autoencoder. Background Art

[0002] In the cloud computing environment, with the explosive growth of data scale and the increasing complexity of computing requirements, anomaly detection as a key security mechanism has become particularly important. Existing anomaly detection methods mainly rely on traditional machine learning and statistical models, which perform well when processing small-scale, static data sets, but often have significant limitations when facing large-scale, dynamic, and distributed data in the cloud computing environment.

[0003] First, traditional anomaly detection methods usually assume that data is stored and processed centrally, which is difficult to achieve in a distributed cloud computing environment. Data in a cloud computing environment is usually distributed and stored on multiple computing nodes, and transmitted and processed over the network. This distributed architecture increases the complexity of data processing, making it difficult for traditional centralized anomaly detection models to be effectively deployed in practical applications. In addition, the data consistency problems brought about by distributed storage and computing also make it difficult for traditional methods to maintain high accuracy and low false alarm rates in a multi-node environment.

[0004] Secondly, existing anomaly detection methods show poor adaptability when processing high-dimensional data. As the complexity of data increases, traditional methods based on rules or simple statistical models find it difficult to capture complex patterns and potential features in the data. This leads to insufficient accuracy and robustness of anomaly detection in practical applications, and it is prone to false positives or false negatives, especially in the face of dynamically changing cloud computing environments. This problem is more prominent.

[0005] In addition, the real-time performance and efficiency of traditional anomaly detection methods are also a major problem. In the cloud computing environment, the demand for real-time processing of data streams is getting higher and higher, and traditional batch processing methods can no longer meet this demand. In the prior art, although some anomaly detection methods based on machine learning can process large-scale data, their training process is usually time-consuming and requires a lot of computing resources. Even after the model training is completed, the real-time detection process still relies on a large amount of computing resource allocation and data transmission, which further increases the response time of the detection system and makes it difficult to meet the real-time detection requirements of high concurrency and high throughput.

[0006] In response to the above-mentioned defects of the existing technology, researchers have proposed some improvement schemes, such as using the autoencoder model in deep learning technology for anomaly detection. However, the traditional autoencoder model is still limited in its effect when facing large-scale distributed data. Although the autoencoder model can capture the characteristics of data by learning the low-dimensional representation of the data, it still shows certain deficiencies when processing high-dimensional and dynamic data in a distributed environment. In addition, when detecting anomalies, traditional autoencoders often rely on a single error metric, which lacks flexibility and adaptability.

[0007] In summary, when processing large-scale, distributed data in cloud computing environments, existing technologies have problems such as difficulty in maintaining data consistency, insufficient high-dimensional data processing capabilities, poor real-time detection performance, and the difficulty of existing VAE models to adapt to distributed environments. These technical defects seriously restrict the effectiveness and practicality of anomaly detection in cloud computing environments. Summary of the invention

[0008] One purpose of the present invention is to propose a storage-computation-training method for anomaly detection in a cloud computing environment based on a variational autoencoder. Through an integrated design, the present invention successfully realizes anomaly detection of large-scale, distributed data in a cloud computing environment.

[0009] According to an embodiment of the present invention, a storage-computation-training integrated method for abnormality detection in a cloud computing environment based on a variational autoencoder includes the following steps:

[0010] S1. Design and initialize a distributed variational autoencoder model based on TensorFlow. The encoder is responsible for encoding the input data into a probability distribution in the latent space, and the decoder is responsible for decoding the variables in the latent space back to the original data space to form reconstructed data.

[0011] S2. In a cloud computing environment, a distributed data storage and computing architecture is constructed, and multiple computing nodes use virtualization technology to manage resources and schedule tasks;

[0012] S3. Deploy and configure the distributed computing framework of TensorFlow on multiple computing nodes. Use its distributed strategy to distribute the training tasks and anomaly detection tasks of the variational autoencoder model on multiple computing nodes. Configure parameter servers and worker computing nodes to enable each computing node to work together in a distributed environment.

[0013] S4. Each computing node uses TensorFlow's distributed strategy to obtain the locally stored preprocessed data set in parallel, execute the forward propagation and backpropagation process of the distributed variational autoencoder model, and use the distributed gradient descent algorithm to synchronously update the model parameters between each computing node. Through the parameter server architecture of TensorFlow, the consistency and synchronization of the model parameters of each computing node are achieved;

[0014] S5. After the training is completed, each computing node uses the trained distributed variational autoencoder model to perform anomaly detection on the real-time input data. Each computing node maps the input data to the latent space through the encoder and generates reconstructed data through the decoder.

[0015] S6. Through the distributed integration strategy of TensorFlow, the anomaly detection results of multiple computing nodes are fused and processed. After each computing node compares the detected reconstruction error with the preset threshold, the data exceeding the threshold is marked as abnormal. The detection results of multiple computing nodes are summarized through integration strategies such as weighted average or voting to obtain a global anomaly detection conclusion.

[0016] Optionally, the S1 specifically includes:

[0017] S11. Using the TensorFlow framework, define the structure of the variational autoencoder model, where the encoder consists of several layers of fully connected neural networks, responsible for mapping the input data X into a probability distribution in the latent space:

[0018] Z~N(μ(X),σ 2 (X));

[0019] Among them, μ(X) and σ(X) are the mean function and standard deviation function, respectively, which represent the distribution characteristics of the input data in the latent space;

[0020] S12, using a distributed parameter initialization method to initialize the network parameters of each layer in the encoder, the parameters include a weight matrix W and a bias vector b, and by distributing the global parameters W and b to multiple computing nodes, parallel initialization is achieved to meet the high concurrency requirements in the cloud computing environment;

[0021] S13. In the decoder part, a distributed back-propagation neural network is used to decode the variable Z in the latent space back to the original data space. The decoder structure is symmetrical to the encoder, and finally outputs the reconstructed data:

[0022]

[0023] Where D(·) represents the mapping function of the decoder;

[0024] S14. Define the improved reconstruction error loss function Combine the input data X with the reconstructed data The mean square error between them is denoted as δ, and the synchronization consistency metric δ between computing nodes in the cloud computing environment is introduced. The synchronization consistency metric is used to ensure that the reconstruction results between computing nodes remain consistent in the distributed computing environment:

[0025]

[0026] Among them, N is the number of data samples, ∥·∥ represents the norm, λ1 is the weight factor, K is the number of computing nodes, δ k represents the deviation between the reconstruction result of the kth computing node and the global synchronization result;

[0027] S15. Define the improved KL divergence loss function When measuring the difference between the potential distribution Z of the encoder output and the standard normal distribution, a potential space distribution consistency constraint between computing nodes in a distributed training environment is introduced to make the potential distribution of each computing node consistent globally:

[0028]

[0029] Where d is the dimension of the latent space, η1 is the weight factor, Δ k represents the deviation between the potential distribution of the kth computational node and the global potential distribution;

[0030] S16. Use the distributed Adam optimization algorithm to train the parameters of the variational autoencoder model. The optimization goal is to minimize the total loss function:

[0031]

[0032] Among them, β1 is a hyperparameter that adjusts the weight between the reconstruction error and the KL divergence.

[0033] Optionally, the S2 specifically includes:

[0034] S21. In a cloud computing environment, virtualization technology is used to build a distributed data storage and computing architecture, which includes multiple virtual computing nodes C k , k = 1, 2, ..., K, where K is the total number of virtual computing nodes, and the virtual computing nodes are connected through the network to achieve distributed processing of data transmission and computing tasks;

[0035] S22, in the virtual computing node C k On the network, a distributed storage system is deployed. Each virtual computing node uses the HDFS distributed file system to implement block storage and management of data. Each data block D i, i = 1, 2, ..., M, redundant storage is performed among multiple virtual computing nodes, where M is the total number of data blocks;

[0036] S23. In the distributed data storage and computing architecture, computing resources are allocated and scheduled through a virtualized resource manager, which dynamically allocates CPU, memory, and storage according to the computing requirements of the task to maximize the computing resource utilization of each virtual computing node, and uses resource isolation technology to prevent resource competition and interference between different tasks;

[0037] S24. Using a distributed task scheduler to distribute the training tasks and anomaly detection tasks of the variational autoencoder model to each computing node, the distributed task scheduler dynamically adjusts the execution order and distribution of tasks based on task priority, node computing power, and network delay factors to optimize the overall computing efficiency and parallelism of task execution;

[0038] S25. Coordinate task execution through the synchronization mechanism between virtual computing nodes. Each virtual computing node regularly exchanges task execution status and intermediate calculation results, and uses the Paxos distributed consistency protocol to perform global consistency control.

[0039] Optionally, the S3 specifically includes:

[0040] S31, on multiple virtual computing nodes C k Deploy and configure the distributed computing framework of TensorFlow on GitHub. The distributed computing framework includes parameter servers and worker computing nodes. The parameter servers are responsible for centralized storage and updating of global model parameters, while the worker computing nodes are responsible for distributed data processing and model calculation.

[0041] S32, initializing the global parameters of the variational autoencoder model on the parameter server, including the weight matrix W, the bias vector b, and the latent space distribution parameters μ and σ:

[0042] W (0) ,b (0) ,μ (0) (X),σ (0) (X);

[0043] Among them, μ (0) (X) and σ (0) (X) is the initial mean and standard deviation function of the input data X. The parameters are distributed to each worker virtual computing node C through a distributed strategy. k ;

[0044] S33. In each worker virtual computing node C kOn the , configure the distributed strategy of TensorFlow, each worker node receives the global model parameters from the parameter server and uses the locally stored data block D i Perform forward propagation and back propagation calculations for the local model:

[0045]

[0046] in, Indicates that in the worker virtual computing node C k The latent variables on (k) is a noise variable sampled from a standard normal distribution, ⊙ represents element-by-element multiplication. Back propagation updates local model parameters by calculating reconstruction error and KL divergence;

[0047] S34. In each worker virtual computing node C k After completing the local model training, calculate the gradient of the local model parameters and pass the gradient information g k Passed back to the parameter server:

[0048]

[0049] in, is the reconstruction error, is the KL divergence, Represents the gradient calculation of the parameter weight matrix W, the bias vector b, and the potential space distribution parameters μ and σ. The parameter server calculates the gradient information g based on the gradient information returned by all worker nodes. k Update global model parameters;

[0050] S35, the process of the parameter server updating the global model parameters is expressed as:

[0051]

[0052] Among them, α2 is the learning rate, t represents the number of iterations, the parameter server distributes the updated global model parameters to all worker computing nodes, and each worker node uses the latest model parameters to continue the next round of local model training, repeating steps S31-S35 until the model training converges.

[0053] Optionally, the S5 specifically includes:

[0054] S51. After the training is completed, each virtual computing node C k Use the trained distributed variational autoencoder model to train the real-time input data stream X i to process;

[0055] S52. Calculate input data X i and reconstructing data The reconstruction error between

[0056]

[0057] Among them, X i,m and Represents the input data X i and reconstructing data The mth feature dimension of , M is the total number of feature dimensions, α m is the weight factor of the mth dimension, which is used to adjust the relative importance of each feature dimension in the reconstruction error calculation; γ m is the nonlinear adjustment factor of the mth dimension, λ m is the wavelength parameter associated with the mth dimension, which is used to introduce nonlinear oscillation components in the reconstruction error and capture potential complex patterns.

[0058] Optionally, the S6 specifically includes:

[0059] S61, each virtual computing node detects the reconstruction error value and the preset anomaly detection threshold τ k After comparison, the data exceeding the threshold value will be Mark as abnormal data, marking result r k (X i ) is expressed as:

[0060]

[0061] Among them, r k (X i ) represents the detection result of the virtual computing node on the input data, 1 represents abnormality and 0 represents normality;

[0062] S62, each virtual computing node will detect the result r k (X i ) is sent to the parameter server, and the parameter server receives the detection results of each node r k (X i ) is weighted averaged to obtain the global detection result R(X i ):

[0063]

[0064] Among them, w k Indicates the weight of the virtual computing node. The weight can be set according to the node's computing power, data processing volume, or the credibility of the detection result;

[0065] S63, according to the global detection result R(X i ) and the global anomaly detection threshold τ globalFor comparison, if R(X i )>τ global , then the data X i Marked as global abnormal data, if R(X i )≤τ global , then it is marked as normal data, and the global anomaly detection threshold τ global Dynamically adjusted by the following formula:

[0066] τ global =μ R +σ R ·λ3;

[0067] Among them, μ R and σ R are the detection results R(X i ), λ3 is the adjustment factor used to control the sensitivity of global anomaly detection;

[0068] S64, the parameter server sends the global anomaly detection result R(X i ) is stored in the global storage system and an anomaly detection report is generated, which includes detailed information of the anomaly data, detection time and detection results of each node.

[0069] The beneficial effects of the present invention are:

[0070] (1) The present invention realizes an integrated design of storage, computing and training by closely combining data storage, computing and model training. In a cloud computing environment, through virtualization technology and distributed computing architecture, each computing node can work together to perform model training and inference tasks under the distributed TensorFlow framework, significantly reducing the overhead of data transmission and task scheduling, and avoiding the performance bottleneck caused by the separation of storage and computing in traditional methods, thereby greatly improving the response speed and computing efficiency of the system. By using a distributed parameter server architecture and a distributed gradient descent algorithm, the consistency of global model parameters is ensured, further enhancing the stability and efficiency of the system.

[0071] (2) The present invention introduces the VAE model into the anomaly detection task in a distributed cloud computing environment. By modeling the latent space of data through VAE, the potential patterns and complex structures of the data can be better captured. The VAE model training and inference process executed in parallel on multiple virtual computing nodes can effectively handle the anomaly detection task of high-dimensional and complex data. At the same time, the latent space distribution of the VAE model is optimized through KL divergence regularization, which makes the model capture the normal data distribution more accurately, thereby having higher accuracy and robustness in detecting abnormal data. Combined with the global and local reconstruction error metrics, the probability of false positives and negatives is further reduced.

[0072] (3) The present invention proposes an improved reconstruction error calculation method. By introducing nonlinear adjustment factors and dynamic threshold adjustment mechanisms, anomaly detection can adapt to the complexity of different feature spaces in cloud computing environments. In the reconstruction error calculation, nonlinear adjustment factors based on feature dimensions are added, so that the system can capture complex patterns and abnormal features in the data. In addition, by combining the threshold dynamic adjustment mechanism with the mean and variance of the latent space distribution, the sensitivity of anomaly detection can be dynamically adjusted according to the distribution characteristics of the data, thereby maintaining high detection performance in different data scenarios. The present invention is suitable for diversified data and dynamically changing features in cloud computing environments, and improves the adaptability and detection accuracy of the system.

[0073] (4) The present invention adopts a distributed integration strategy. By weighted averaging and global consistency processing the detection results of multiple virtual computing nodes, the reliability and consistency of the final anomaly detection conclusion are ensured. In a cloud computing environment, the computing power and data processing volume of each node may be different. By setting node weights and dynamically adjusting the global threshold, the detection results of each node can be effectively integrated to avoid the impact of false positives or false negatives of a single node on the overall detection results. This not only improves the accuracy of anomaly detection, but also enhances the fault tolerance and reliability of the system in a distributed environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0075] Figure 1 This is a flow chart of a storage-computation-training method for abnormality detection in a cloud computing environment based on a variational autoencoder proposed by the present invention;

[0076] Figure 2 This is a schematic diagram of the distributed storage and computing architecture of data in a cloud computing environment in a storage-computation-training integrated method for anomaly detection in a cloud computing environment based on a variational autoencoder proposed by the present invention. DETAILED DESCRIPTION

[0077] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0078] refer to Figure 1-2 , a storage-computation-training method for abnormality detection in a cloud computing environment based on a variational autoencoder, comprising the following steps:

[0079] S1. Design and initialize a distributed variational autoencoder model based on TensorFlow. The encoder is responsible for encoding the input data into a probability distribution in the latent space, and the decoder is responsible for decoding the variables in the latent space back to the original data space to form reconstructed data.

[0080] S2. In a cloud computing environment, a distributed data storage and computing architecture is constructed, and multiple computing nodes use virtualization technology to manage resources and schedule tasks;

[0081] S3. Deploy and configure the distributed computing framework of TensorFlow on multiple computing nodes. Use its distributed strategy to distribute the training tasks and anomaly detection tasks of the variational autoencoder model on multiple computing nodes. Configure parameter servers and worker computing nodes to enable each computing node to work together in a distributed environment.

[0082] S4. Each computing node uses TensorFlow's distributed strategy to obtain the locally stored preprocessed data set in parallel, execute the forward propagation and backpropagation process of the distributed variational autoencoder model, and use the distributed gradient descent algorithm to synchronously update the model parameters between each computing node. Through the parameter server architecture of TensorFlow, the consistency and synchronization of the model parameters of each computing node are achieved;

[0083] S5. After the training is completed, each computing node uses the trained distributed variational autoencoder model to perform anomaly detection on the real-time input data. Each computing node maps the input data to the latent space through the encoder and generates reconstructed data through the decoder.

[0084] S6. Through the distributed integration strategy of TensorFlow, the anomaly detection results of multiple computing nodes are fused and processed. After each computing node compares the detected reconstruction error with the preset threshold, the data exceeding the threshold is marked as abnormal. The detection results of multiple computing nodes are summarized through integration strategies such as weighted average or voting to obtain a global anomaly detection conclusion.

[0085] In this implementation, S1 specifically includes:

[0086] S11. Using the TensorFlow framework, define the structure of the variational autoencoder model, where the encoder consists of several layers of fully connected neural networks, responsible for mapping the input data X into a probability distribution in the latent space:

[0087] Z~N(μ(X),σ 2 (X));

[0088] Among them, μ(X) and σ(X) are the mean function and standard deviation function, respectively, which represent the distribution characteristics of the input data in the latent space;

[0089] S12, using a distributed parameter initialization method to initialize the network parameters of each layer in the encoder, the parameters include a weight matrix W and a bias vector b, and by distributing the global parameters W and b to multiple computing nodes, parallel initialization is achieved to meet the high concurrency requirements in the cloud computing environment;

[0090] S13. In the decoder part, a distributed back-propagation neural network is used to decode the variable Z in the latent space back to the original data space. The decoder structure is symmetrical to the encoder, and finally outputs the reconstructed data:

[0091]

[0092] Where D(·) represents the mapping function of the decoder;

[0093] S14. Define the improved reconstruction error loss function Combine the input data X with the reconstructed data The mean square error between them is denoted as δ, and the synchronization consistency metric δ between computing nodes in the cloud computing environment is introduced. The synchronization consistency metric is used to ensure that the reconstruction results between computing nodes remain consistent in the distributed computing environment:

[0094]

[0095] Among them, N is the number of data samples, ∥·∥ represents the norm, λ1 is the weight factor, K is the number of computing nodes, δ k represents the deviation between the reconstruction result of the kth computing node and the global synchronization result;

[0096] S15. Define the improved KL divergence loss function When measuring the difference between the potential distribution Z of the encoder output and the standard normal distribution, a potential space distribution consistency constraint between computing nodes in a distributed training environment is introduced to make the potential distribution of each computing node consistent globally:

[0097]

[0098] Where d is the dimension of the latent space, η1 is the weight factor, Δ k represents the deviation between the potential distribution of the kth computational node and the global potential distribution;

[0099] S16. Use the distributed Adam optimization algorithm to train the parameters of the variational autoencoder model. The optimization goal is to minimize the total loss function:

[0100]

[0101] Among them, β1 is a hyperparameter that adjusts the weight between the reconstruction error and the KL divergence.

[0102] In this implementation, S2 specifically includes:

[0103] S21. In a cloud computing environment, virtualization technology is used to build a distributed data storage and computing architecture, which includes multiple virtual computing nodes C k , k = 1, 2, ..., K, where K is the total number of virtual computing nodes, and the virtual computing nodes are connected through the network to achieve distributed processing of data transmission and computing tasks;

[0104] S22, on virtual computing node C k On the network, a distributed storage system is deployed. Each virtual computing node uses the HDFS distributed file system to implement block storage and management of data. Each data block D i , i = 1, 2, ..., M, redundant storage is performed among multiple virtual computing nodes, where M is the total number of data blocks;

[0105] S23. In the distributed data storage and computing architecture, computing resources are allocated and scheduled through a virtualized resource manager, which dynamically allocates CPU, memory, and storage according to the computing requirements of the task to maximize the computing resource utilization of each virtual computing node, and uses resource isolation technology to prevent resource competition and interference between different tasks;

[0106] S24. Using a distributed task scheduler to distribute the training tasks and anomaly detection tasks of the variational autoencoder model to each computing node, the distributed task scheduler dynamically adjusts the execution order and distribution of tasks based on task priority, node computing power, and network delay factors to optimize the overall computing efficiency and parallelism of task execution;

[0107] S25. Coordinate task execution through the synchronization mechanism between virtual computing nodes. Each virtual computing node regularly exchanges task execution status and intermediate calculation results, and uses the Paxos distributed consistency protocol to perform global consistency control.

[0108] In this implementation, S3 specifically includes:

[0109] S31, on multiple virtual computing nodes C k Deploy and configure the distributed computing framework of TensorFlow on GitHub. The distributed computing framework includes parameter servers and worker computing nodes. The parameter servers are responsible for centralized storage and updating of global model parameters, while the worker computing nodes are responsible for distributed data processing and model calculation.

[0110] S32. Initialize the global parameters of the variational autoencoder model on the parameter server, including the weight matrix W, the bias vector b, and the latent space distribution parameters μ and σ:

[0111] W (0) ,b (0) ,μ (0) (X),σ (0) (X);

[0112] Among them, μ (0) (X) and σ (0) (X) is the initial mean and standard deviation function of the input data X. The parameters are distributed to each worker virtual computing node C through a distributed strategy. k ;

[0113] S33. In each worker virtual computing node C k On the , configure the distributed strategy of TensorFlow, each worker node receives the global model parameters from the parameter server and uses the locally stored data block D i Perform forward propagation and back propagation calculations for the local model:

[0114]

[0115] in, Indicates that in the worker virtual computing node C k The latent variables on (k) is a noise variable sampled from a standard normal distribution, ⊙ represents element-by-element multiplication. Back propagation updates local model parameters by calculating reconstruction error and KL divergence;

[0116] S34. In each worker virtual computing node C k After completing the local model training, calculate the gradient of the local model parameters and pass the gradient information g k Passed back to the parameter server:

[0117]

[0118] in, is the reconstruction error, is the KL divergence, Represents the gradient calculation of the parameter weight matrix W, the bias vector b, and the potential space distribution parameters μ and σ. The parameter server calculates the gradient information g based on the gradient information returned by all worker nodes. k Update global model parameters;

[0119] S35, the process of the parameter server updating the global model parameters is expressed as:

[0120]

[0121]

[0122] Among them, α2 is the learning rate, t represents the number of iterations, the parameter server distributes the updated global model parameters to all worker computing nodes, and each worker node uses the latest model parameters to continue the next round of local model training, repeating steps S31-S35 until the model training converges.

[0123] In this implementation, S5 specifically includes:

[0124] S51. After the training is completed, each virtual computing node C k Use the trained distributed variational autoencoder model to train the real-time input data stream X i to process;

[0125] S52. Calculate input data X i and reconstructing data The reconstruction error between

[0126]

[0127] Among them, X i,m and Represents the input data X i and reconstructing data The mth feature dimension of , M is the total number of feature dimensions, α m is the weight factor of the mth dimension, which is used to adjust the relative importance of each feature dimension in the reconstruction error calculation; γ m is the nonlinear adjustment factor of the mth dimension, λ m is the wavelength parameter associated with the mth dimension, which is used to introduce nonlinear oscillation components in the reconstruction error and capture potential complex patterns.

[0128] In this implementation, S6 specifically includes:

[0129] S61, each virtual computing node detects the reconstruction error value and the preset anomaly detection threshold τ k After comparison, the data exceeding the threshold value will be Mark as abnormal data, marking result r k (X i ) is expressed as:

[0130]

[0131] Among them, r k (X i) represents the detection result of the virtual computing node on the input data, 1 represents abnormality and 0 represents normality;

[0132] S62, each virtual computing node will detect the result r k 9X i ) is sent to the parameter server, and the parameter server receives the detection results of each node r k 9X i ) is weighted averaged to obtain the global detection result R(X i ):

[0133]

[0134] Among them, w k Indicates the weight of the virtual computing node. The weight can be set according to the node's computing power, data processing volume, or the credibility of the detection result;

[0135] S63, according to the global detection result R(X i ) and the global anomaly detection threshold τ global For comparison, if R(X i )>τ global , then the data X i Marked as global abnormal data, if R(X i )≤τ global , then it is marked as normal data, and the global anomaly detection threshold τ global Dynamically adjusted by the following formula:

[0136] τ global =μ R +σ R ·λ3;

[0137] Among them, μ R and σ R are the detection results R(X i ), λ3 is the adjustment factor used to control the sensitivity of global anomaly detection;

[0138] S64, the parameter server sends the global anomaly detection result R(X i ) is stored in the global storage system and an anomaly detection report is generated, which includes detailed information of the anomaly data, detection time and detection results of each node.

[0139] Embodiment 1:

[0140] A large financial institution relies on cloud computing platforms to process a large amount of transaction data and user behavior data in its global business. Financial institutions process millions of transactions every day and generate more than thousands of log data every second. These data need to be distributed and processed in multiple data centers around the world. In order to ensure the security of the system and the integrity of the data, the financial institution introduced the anomaly detection storage, computing and training integrated method in a cloud computing environment based on a variational autoencoder of the present invention to monitor the operating status of the system in real time, detect and respond to potential network attacks and abnormal behaviors.

[0141] In Example 1, the data center of a financial institution processes millions of financial transactions and user behavior data simultaneously around the world. The peak hours every day are from 12 noon to 2 pm, with huge data traffic, and the anomaly detection pressure faced by the system is also at its highest. One day at noon, the system monitoring module suddenly discovered an abnormal phenomenon: the host with the source IP address of "192.168.50.23" initiated a large number of database query requests in a short period of time, and its packet length, query frequency, and query content characteristics all showed abnormalities.

[0142] According to the method of the present invention, virtual computing nodes distributed in different data centers immediately process these log data. First, the virtual computing nodes input these log data into the pre-trained variational autoencoder model, and the encoder part of the model maps these data to the latent space. The system noticed that the distribution of the latent variables was significantly different from the normal query behavior, and the reconstruction error between the reconstructed data generated by the decoder and the original data far exceeded the normal threshold of the system.

[0143] Specifically, the host with source IP address "192.168.50.23" initiated more than 10,000 query requests in just 30 seconds, while under normal circumstances, the average query request in the same time period is 500. The packet length of these abnormal requests is greater than 1024 bytes, and the query content involves sensitive information, such as user passwords, account balances, etc. Through the latent space mapping analysis of the variational autoencoder, the host's abnormal score reached 0.92, while the threshold set by the system is 0.75.

[0144] The system generated a detailed attack report, including detailed information about the IP address, feature analysis of abnormal requests, reconstruction error values, and the degree of deviation of the potential space distribution. The report showed that the abnormal behavior of the IP address was highly similar to the known SQL injection attack pattern, and it was very likely that hackers attempted to attack the database through the IP address. The system immediately triggered an early warning mechanism and sent the information to the network security team in real time.

[0145] After receiving the warning signal, the security team immediately took countermeasures. They first blocked the connection of the host "192.168.50.23" to prevent further malicious behavior. Then, through a detailed analysis of the attack report generated by the system, it was confirmed that this was a highly concealed SQL injection attack attempt. Based on the abnormal characteristics in the report, the team further strengthened the system's protection measures, fixed possible security vulnerabilities, and reported the detailed information of the attack to a higher-level security monitoring system.

[0146] In this incident, the system completed the entire process from data collection, anomaly detection to early warning within 20 seconds, greatly shortening the response time and ensuring the data security of financial institutions. Compared with the traditional rule-based anomaly detection method, the method of the present invention not only improves the accuracy of detection, but also greatly reduces the false alarm rate. In similar situations in the past, traditional systems may miss such attacks because the abnormal features are not obvious, resulting in serious consequences.

[0147] In addition, in another data center, the system detected another set of abnormal request traffic initiated by the IP address "172.16.34.100". The host accessed multiple user accounts in a short period of time and attempted to reset passwords multiple times. The system's variational autoencoder model identified abnormal latent spatial distribution of these requests, with a reconstruction error value of 0.88, while the normal reconstruction error value is usually less than 0.6. After analysis, the system determined that these requests were consistent with a typical brute force attack pattern.

[0148] Similarly, the system quickly generated an anomaly report, recording the detailed behavioral characteristics of the IP address, the request content, and the timeline. The security team immediately blacklisted the IP address and tracked it in detail, ultimately confirming that it was a distributed brute force cracking attempt. The team further improved the system's ability to detect similar attacks by adjusting the model's threshold and reconstruction error calculation method.

[0149] In actual applications, financial institutions compared the performance of the method of the present invention with their traditional rule-based anomaly detection systems in handling the above events. The comparison data is shown in Table 1 below:

[0150] Table 1 Performance data of the method of the present invention and its traditional method in processing events

[0151]

[0152]

[0153] Through the above cases and data comparison, it can be seen that the application of the method of the present invention in the actual cloud computing environment not only effectively solves the shortcomings of traditional anomaly detection methods, but also significantly improves the detection efficiency and accuracy, especially when processing high concurrency and complex data, and provides a strong guarantee for the network security of financial institutions.

[0154] The present invention realizes the integrated design of storage, computing and training by closely combining data storage, computing and model training. In the cloud computing environment, through virtualization technology and distributed computing architecture, each computing node can work together to perform model training and inference tasks under the distributed TensorFlow framework, which significantly reduces the overhead of data transmission and task scheduling, avoids the performance bottleneck caused by the separation of storage and computing in traditional methods, thereby greatly improving the response speed and computing efficiency of the system. By using a distributed parameter server architecture and a distributed gradient descent algorithm, the consistency of global model parameters is ensured, further enhancing the stability and efficiency of the system.

[0155] The present invention introduces the VAE model into the anomaly detection task in a distributed cloud computing environment. By modeling the latent space of data through VAE, the potential patterns and complex structures of the data can be better captured. The VAE model training and inference process executed in parallel on multiple virtual computing nodes can effectively handle the anomaly detection tasks of high-dimensional and complex data. At the same time, the latent space distribution of the VAE model is optimized through KL divergence regularization, so that the model can capture the normal data distribution more accurately, thereby having higher accuracy and robustness when detecting abnormal data. Combined with the global and local reconstruction error metrics, the probability of false positives and false negatives is further reduced.

[0156] The present invention proposes an improved reconstruction error calculation method. By introducing a nonlinear adjustment factor and a dynamic threshold adjustment mechanism, anomaly detection can adapt to the complexity of different feature spaces in a cloud computing environment. In the reconstruction error calculation, a nonlinear adjustment factor based on the feature dimension is added, so that the system can capture complex patterns and abnormal features in the data. In addition, by combining the threshold dynamic adjustment mechanism with the mean and variance of the latent space distribution, the sensitivity of anomaly detection can be dynamically adjusted according to the distribution characteristics of the data, thereby maintaining a high detection performance in different data scenarios. The present invention is suitable for diversified data and dynamically changing features in a cloud computing environment, and improves the adaptability and detection accuracy of the system.

[0157] The present invention adopts a distributed integration strategy, and ensures the reliability and consistency of the final anomaly detection conclusion by weighted averaging and global consistency processing of the detection results of multiple virtual computing nodes. In a cloud computing environment, the computing power and data processing volume of each node may be different. By setting node weights and dynamically adjusting the global threshold, the detection results of each node can be effectively integrated, avoiding the impact of false positives or false negatives of a single node on the overall detection results. This not only improves the accuracy of anomaly detection, but also enhances the fault tolerance and reliability of the system in a distributed environment.

[0158] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A storage-computation-training method for abnormality detection in cloud computing environment based on variational autoencoder, characterized in that: The steps include: S1. Design and initialize a distributed variational autoencoder model based on TensorFlow. The encoder is responsible for encoding the input data into a probability distribution in the latent space, and the decoder is responsible for decoding the variables in the latent space back to the original data space to form reconstructed data. S2. In a cloud computing environment, a distributed data storage and computing architecture is constructed, and multiple computing nodes use virtualization technology to manage resources and schedule tasks; S3. Deploy and configure the distributed computing framework of TensorFlow on multiple computing nodes. Use its distributed strategy to distribute the training tasks and anomaly detection tasks of the variational autoencoder model on multiple computing nodes, and configure parameter servers and worker computing nodes. S4. Each computing node uses TensorFlow's distributed strategy to obtain the locally stored preprocessed data set in parallel, executes the forward propagation and backpropagation process of the distributed variational autoencoder model, and uses the distributed gradient descent algorithm to synchronously update the model parameters among the computing nodes; S5. After the training is completed, each computing node uses the trained distributed variational autoencoder model to perform anomaly detection on the real-time input data. Each computing node maps the input data to the latent space through the encoder and generates reconstructed data through the decoder. S6. Through the distributed integration strategy of TensorFlow, the anomaly detection results of multiple computing nodes are fused and processed. After each computing node compares the detected reconstruction error with the preset threshold, the data exceeding the threshold is marked as abnormal. The detection results of multiple computing nodes are summarized through integration strategies such as weighted average or voting to obtain a global anomaly detection conclusion.

2. According to claim 1, a storage-computation-training integrated method for abnormality detection in a cloud computing environment based on a variational autoencoder is characterized in that: The S1 specifically includes: S11. Using the TensorFlow framework, define the structure of the variational autoencoder model, where the encoder consists of several layers of fully connected neural networks, responsible for mapping the input data X into a probability distribution in the latent space: Z~N(μ(X),σ 2 (X)); Among them, μ(X) and σ(X) are the mean function and standard deviation function, respectively, which represent the distribution characteristics of the input data in the latent space; S12, using a distributed parameter initialization method to initialize the network parameters of each layer in the encoder, the parameters include a weight matrix W and a bias vector b, and by distributing the global parameters W and b to multiple computing nodes, parallel initialization is achieved to meet the high concurrency requirements in the cloud computing environment; S13. In the decoder part, a distributed back-propagation neural network is used to decode the variable Z in the latent space back to the original data space. The decoder structure is symmetrical to the encoder, and finally outputs the reconstructed data: Where D(·) represents the mapping function of the decoder; S14. Define the improved reconstruction error loss function Combine the input data X with the reconstructed data The mean square error between them is denoted as δ, and the synchronization consistency metric δ between computing nodes in the cloud computing environment is introduced. The synchronization consistency metric is used to ensure that the reconstruction results between computing nodes remain consistent in the distributed computing environment: Among them, N is the number of data samples, ∥·∥ represents the norm, λ1 is the weight factor, K is the number of computing nodes, δ k represents the deviation between the reconstruction result of the kth computing node and the global synchronization result; S15. Define the improved KL divergence loss function When measuring the difference between the potential distribution Z of the encoder output and the standard normal distribution, a potential space distribution consistency constraint between computing nodes in a distributed training environment is introduced to make the potential distribution of each computing node consistent globally: Where d is the dimension of the latent space, η1 is the weight factor, Δ k represents the deviation between the potential distribution of the kth computational node and the global potential distribution; S16. Use the distributed Adam optimization algorithm to train the parameters of the variational autoencoder model. The optimization goal is to minimize the total loss function: Among them, β1 is a hyperparameter that adjusts the weight between the reconstruction error and the KL divergence.

3. According to claim 1, a storage-computation-training integrated method for abnormality detection in a cloud computing environment based on a variational autoencoder is characterized in that: The S2 specifically includes: S21. In a cloud computing environment, virtualization technology is used to build a distributed data storage and computing architecture, which includes multiple virtual computing nodes C k , k = 1, 2, ..., K, where K is the total number of virtual computing nodes, and the virtual computing nodes are connected through the network to achieve distributed processing of data transmission and computing tasks; S22, in the virtual computing node C k On the network, a distributed storage system is deployed. Each virtual computing node uses the HDFS distributed file system to implement block storage and management of data. Each data block D i , i = 1, 2, ..., M, redundant storage is performed among multiple virtual computing nodes, where M is the total number of data blocks; S23. In the distributed data storage and computing architecture, computing resources are allocated and scheduled through a virtualized resource manager, which dynamically allocates CPU, memory, and storage according to the computing requirements of the task to maximize the computing resource utilization of each virtual computing node, and uses resource isolation technology to prevent resource competition and interference between different tasks; S24. Using a distributed task scheduler to distribute the training tasks and anomaly detection tasks of the variational autoencoder model to each computing node, the distributed task scheduler dynamically adjusts the execution order and distribution of tasks based on task priority, node computing power, and network delay factors to optimize the overall computing efficiency and parallelism of task execution; S25. Coordinate task execution through the synchronization mechanism between virtual computing nodes. Each virtual computing node regularly exchanges task execution status and intermediate calculation results, and uses the Paxos distributed consistency protocol to perform global consistency control.

4. According to claim 1, a storage-computation-training integrated method for abnormality detection in a cloud computing environment based on a variational autoencoder is characterized in that: The S3 specifically includes: S31, on multiple virtual computing nodes C k Deploy and configure the distributed computing framework of TensorFlow on GitHub. The distributed computing framework includes parameter servers and worker computing nodes. The parameter servers are responsible for centralized storage and updating of global model parameters, while the worker computing nodes are responsible for distributed data processing and model calculation. S32, initializing the global parameters of the variational autoencoder model on the parameter server, including the weight matrix W, the bias vector b, and the latent space distribution parameters μ and σ: W (0) ,b (0) ,m (0) (X),s (0) (X); Among them, μ (0) (X) and σ (0) (X) is the initial mean and standard deviation function of the input data X. The parameters are distributed to each worker virtual computing node C through a distributed strategy. k ; S33. In each worker virtual computing node C k On the , configure the distributed strategy of TensorFlow, each worker node receives the global model parameters from the parameter server and uses the locally stored data block D i Perform forward propagation and back propagation calculations for the local model: in, Indicates that in the worker virtual computing node C k The latent variables on (k) is a noise variable sampled from a standard normal distribution, ⊙ represents element-by-element multiplication. Back propagation updates local model parameters by calculating reconstruction error and KL divergence; S34. In each worker virtual computing node C k After completing the local model training, calculate the gradient of the local model parameters and pass the gradient information g k Passed back to the parameter server: in, is the reconstruction error, is the KL divergence, Represents the gradient calculation of the parameter weight matrix W, the bias vector b, and the potential space distribution parameters μ and σ. The parameter server calculates the gradient information g based on the gradient information returned by all worker nodes. k Update global model parameters; S35, the process of the parameter server updating the global model parameters is expressed as: Among them, α2 is the learning rate, t represents the number of iterations, the parameter server distributes the updated global model parameters to all worker computing nodes, and each worker node uses the latest model parameters to continue the next round of local model training, repeating steps S31-S35 until the model training converges.

5. According to claim 1, a storage-computation-training integrated method for abnormality detection in a cloud computing environment based on a variational autoencoder is characterized in that: The S5 specifically includes: S51. After the training is completed, each virtual computing node C k Use the trained distributed variational autoencoder model to train the real-time input data stream X i to process; S52. Calculate input data X i and reconstructing data The reconstruction error between Among them, X i,m and Represents the input data X i and reconstructing data The mth feature dimension of , M is the total number of feature dimensions, α m is the weight factor of the mth dimension, which is used to adjust the relative importance of each feature dimension in the reconstruction error calculation; γ m is the nonlinear adjustment factor of the mth dimension, λ m is the wavelength parameter associated with the mth dimension, which is used to introduce nonlinear oscillation components in the reconstruction error and capture potential complex patterns.

6. According to claim 1, a storage-computation-training integrated method for abnormality detection in a cloud computing environment based on a variational autoencoder is characterized in that: The S6 specifically includes: S61, each virtual computing node detects the reconstruction error value and the preset anomaly detection threshold τ k After comparison, the data exceeding the threshold value will be Mark as abnormal data, marking result r k (X i ) is expressed as: Among them, r k (X i ) represents the detection result of the virtual computing node on the input data, 1 represents abnormality and 0 represents normality; S62, each virtual computing node will detect the result r k (X i ) is sent to the parameter server, and the parameter server receives the detection results of each node r k (X i ) is weighted averaged to obtain the global detection result R(X i ): Among them, w k Indicates the weight of the virtual computing node. The weight can be set according to the node's computing power, data processing volume, or the credibility of the detection result; S63, according to the global detection result R(X i ) and the global anomaly detection threshold τ global For comparison, if R(X i )>τ global , then the data X i Marked as global abnormal data, if R(X i )≤τ global , then it is marked as normal data, and the global anomaly detection threshold τ global Dynamically adjusted by the following formula: t global =μ R +s R ·λ3; Among them, μ R and σ R are the detection results R(X i ), λ3 is the adjustment factor used to control the sensitivity of global anomaly detection; S64, the parameter server sends the global anomaly detection result R(X i ) is stored in the global storage system and an anomaly detection report is generated, which includes detailed information of the anomaly data, detection time and detection results of each node.

Images