Big data processing system based on AI cloud computing training and computer device

Abstract

The invention relates to the field of big data processing, in particular to a big data processing system based on AI cloud computing training and a computer device. The system comprises a data acquisition module, a preprocessing module, a data storage optimization module, a machine learning processing module, a deep learning classification module, a real-time data processing module, a data quality guarantee module and a user interface and configuration management module. The output end of the data acquisition module is connected with the input ends of the preprocessing module and the machine learning processing module, and the input end of the deep learning classification module is connected with the output end of the machine learning processing module. The output end of the deep learning classification module is connected with the input end of the real-time data processing module and the input end of the user interface and configuration management module, the input end of the data storage optimization module is connected with the output end of the preprocessing module, and the output end of the preprocessing module is connected with the input end of the data quality guarantee module. The method is suitable for big data processing.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a big data processing system and a computer device based on AI cloud computing training. Background Art

[0002] The big data collection and cleaning system is a set of technical solutions designed specifically for processing and analyzing massive data. The key lies in effectively collecting data from diverse data sources and preprocessing it to ensure data quality and availability. However, traditional data collection methods cannot effectively cope with diverse and high-flow data sources, and in large-scale data environments, manually identifying and correcting data quality issues (such as missing values, outliers, format errors, etc.) is time-consuming and error-prone. At the same time, when high-precision classification of complex data is required, the current big data processing algorithms may not be sufficient to meet the needs, resulting in data flow congestion and increased processing delays, which in turn affects the performance and response speed of the entire system. In addition, the classification and prediction of complex data based on the current big data processing algorithms cannot achieve the required accuracy and cover all situations. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a big data processing system and computer device based on AI cloud computing training, which improves data processing efficiency and improves the accuracy and reliability of big data anomaly detection.

[0004] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. In a first aspect, the present invention provides a big data processing system based on AI cloud computing training, comprising: It includes data acquisition module, preprocessing module, data storage optimization module, machine learning processing module, deep learning classification module, real-time data processing module, data quality assurance module, and user interface and configuration management module; The data acquisition module is used to collect data and send the collected data to the preprocessing module and the machine learning processing module respectively. The preprocessing module uses AI to identify abnormal data and irregular data, and then sends the data to the data storage optimization module. The data storage optimization module partitions and stores the data by type. The machine learning processing module uses a method combining support vector machine and adaptive threshold dynamic adjustment to process the data, and then sends the processed data to the deep learning classification module. The deep learning classification module uses a convolutional neural network deep learning model to process the data, and then sends the processed data to the user interface and configuration management module and the real-time data processing module respectively. The user interface and configuration management module update the data and analysis results in real time according to the data processed by the deep learning classification module, and support user-defined queries and report generation. The real-time data processing module uses a real-time anomaly detection algorithm and dynamic threshold adjustment to process real-time data anomalies, and then sends the processed data to the data quality assurance module. The data quality assurance module configures data verification rules, routine error checks, and repairs or reports detected problems according to the data processed by the real-time data processing module and the data stored in the data storage optimization module; The preprocessing module uses AI to identify abnormal data and irregular data, and adopts an outlier detection algorithm, specifically including: Clustering-based anomaly detection: Set two parameters: ε and MinPts, MinPts is the minimum number of samples, and ε represents the maximum distance from any given point to its neighboring points; For each point, calculate the number of samples in its ε-neighborhood; If a point has at least MinPts points in its ε-neighborhood, then the point is marked as a core point, otherwise it is a non-core point; Among non-core points, if a point does not belong to the ε-neighborhood of any other core point, it is considered an outlier, as follows: ; where Nϵ(p) is the set of points in the ε-neighborhood of point p, D is the entire dataset, and dist(p,q) measures the distance between points p and q; The machine learning processing module adopts a method combining support vector machine and adaptive threshold dynamic adjustment, which is as follows: Model building and training: Decision function: ; Where w is the weight vector, ϕ(x) is the input data mapped to the high-dimensional space, and b is the bias term; Optimization problem: ; Where C is the regularization parameter, which controls the smoothness of the misclassification and decision surface; L is the loss function, which measures the gap between the model prediction value and the true label; The machine learning processing module sends the processed data to the deep learning classification module; The deep learning classification module adopts a convolutional neural network deep learning model, which is as follows: The convolutional layer extracts local features: Layer definition: ; in, is the convolution kernel, ∗ represents the convolution operation, is the bias, σ is the activation function, and x is the input; The pooling layer reduces the feature dimension: Max Pooling: ; Fully connected layer decision output: All features are flattened into a vector and fully connected: ; W and b are the weight matrix and bias vector respectively; Output layer classification prediction: The vector z is converted into a probability distribution p through the Softmax function, where each element Indicates the probability that the input belongs to the kth category, and uses the Softmax function to make probability predictions for multiple categories: ; is the raw score of the kth class; e is the base of natural logarithms; is the exponential sum of the raw scores of all categories, used for normalization to ensure that the sum of all probabilities is 1; is the output passed from the previous layer; Introducing the attention mechanism to strengthen the convolutional neural network deep learning model's focus on important features: Global average pooling: Perform global average pooling on the feature map of each channel to obtain the feature vector of each channel: ; Where F(i,j,c) is the eigenvalue of the cth channel at position (i,j), H and W are the height and width of the feature map; Global maximum pooling: Perform global maximum pooling on the feature map of each channel to obtain the feature vector of each channel: ; Shared multi-layer perceptron: The results of GAP and GMP are transformed through a shared multi-layer perceptron (MLP) to obtain two intermediate feature vectors: ; ; Where W1 and W2 are the weight matrices of the MLP, and σ is the activation function; Feature fusion: Add the two intermediate feature vectors to get the final channel attention weight: ; Weight application, apply the channel attention weights to the original feature map: ; Represents the feature map after applying the channel attention weight; The deep learning classification module sends the processed data to the real-time data processing module; The real-time data processing module uses a real-time anomaly detection algorithm and dynamic threshold adjustment to process real-time data anomaly points, which are specifically as follows: Moving window mean and standard deviation: For each new data point , maintain a fixed-size window W for calculation: ; ; Among them, μt and σt represent the mean and standard deviation within the window respectively; Z-score calculation: For each new data point , calculate its Z-score: ; When ∣ is greater than a certain threshold, then it is considered It is an outlier.

[0005] Furthermore, the specific functions of the preprocessing module also include: Anomaly Detection Based on Sparse Representation: Use sparse coding to learn a low-dimensional representation of the data. Assuming that the data is represented by a sparse linear combination of basis vectors, while the abnormal data cannot be represented, a dictionary E is constructed, where each column is a basis element; For each data point , the optimal sparse code αi is found by solving the following optimization problem: ; Calculate the reconstruction error ( , a high reconstruction error indicates It is abnormal; The optimization problem is the Lasso regression problem, where λ controls the sparsity level, is the L1 norm (sparse regularization term).

[0006] Furthermore, the specific functions of the preprocessing module also include: Context-aware time series anomaly detection: Data preparation: Time series data x(t): original time series data; Context information C(t): contains external environmental factors and historical data pattern information; Frequency domain representation X(f): The frequency domain representation obtained by performing a fast Fourier transform (FFT) on x(t); Frequency Domain Analysis: Apply FFT transform to the time series data x(t) to obtain the frequency domain representation X(f): , j represents the imaginary unit, f Indicates frequency; Contextual Information Fusion: The context information C(t) is integrated into the frequency domain representation X(f) to form a context-aware frequency domain representation XC(f): ; Among them, WC(f) is the weight function of context information, which can be adjusted according to the importance of context information; Anomaly Detection: The anomaly detection is performed on the context-aware frequency domain representation XC(f) as follows: Calculate the amplitude: ; Determine abnormal frequency: Set a threshold T. If the amplitude A(f) of a certain frequency f is higher than the background noise level, the moment corresponding to the frequency is considered abnormal: ; Where μA and σA are the mean and standard deviation of the amplitude A(f), respectively, and k is an adjustable parameter; Abnormal point location: For each time point t, check whether its corresponding frequency domain representation XC(f) has a frequency component whose amplitude exceeds the threshold T. If so, the data at time t is considered abnormal.

[0007] Furthermore, the dynamic threshold adjustment specifically includes: ; in, is the dynamic threshold at the current moment, is the dynamic threshold of the previous moment, λ is the forget factor, which ranges from 0 to 1 and is used to balance the influence of the historical threshold and the latest Z-score; It is the absolute value of the Z-score at the current moment.

[0008] In a second aspect, the present invention provides a computer device, including a memory, wherein the memory stores program instructions, and when the program instructions are run, the big data processing method based on AI cloud computing training as described above is executed.

[0009] The beneficial effects of the present invention are: The present invention integrates multiple functional modules such as data acquisition, preprocessing, storage optimization, machine learning processing, deep learning classification, real-time data processing, data quality assurance, user interface and configuration management into one system. The integrated design not only improves the efficiency of data processing, but also enhances the overall coordination and ease of use of the system. The adaptive learning mechanism introduced in the machine learning processing module can automatically adjust the model parameters according to the data characteristics, which not only improves the generalization ability of the model, but also enhances the adaptability of the model to different application scenarios, so that the system can maintain high efficiency and accuracy in a constantly changing data environment. By adding context-aware time series anomaly detection, it takes into account the context information of the time series data, and improves the accuracy and reliability of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a structural block diagram of a big data processing system based on AI cloud computing training provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0012] The present invention provides a big data processing system based on AI cloud computing training, such as Figure 1As shown, it includes a data acquisition module, a preprocessing module, a data storage optimization module, a machine learning processing module, a deep learning classification module, a real-time data processing module, a data quality assurance module, and a user interface and configuration management module. The output end of the data acquisition module is respectively connected to the input end of the preprocessing module and the machine learning processing module, the input end of the deep learning classification module is connected to the output end of the machine learning processing module, the output end of the deep learning classification module is respectively connected to the input end of the real-time data processing module and the input end of the user interface and the configuration management module, the input end of the data storage optimization module is connected to the output end of the preprocessing module, and the output end of the preprocessing module is connected to the input end of the data quality assurance module; The data acquisition module is used to collect data and send the collected data to the preprocessing module and the machine learning processing module respectively. The preprocessing module uses AI to identify abnormal data and irregular data, and then sends the data to the data storage optimization module. The data storage optimization module partitions and stores the data by type. The machine learning processing module uses a method combining support vector machine and adaptive threshold dynamic adjustment to process the data, and then sends the processed data to the deep learning classification module. The deep learning classification module uses a convolutional neural network deep learning model to process the data, and then sends the processed data to the user interface and configuration management module and the real-time data processing module respectively. The user interface and the configuration management module update the data and analysis results in real time according to the data processed by the deep learning classification module, and support user-defined queries and report generation. The real-time data processing module uses a real-time anomaly detection algorithm and dynamic threshold adjustment to process real-time data anomalies, and then sends the processed data to the data quality assurance module. The data quality assurance module configures data verification rules, routine error checks, and repairs or reports detected problems according to the data processed by the real-time data processing module and the data stored in the data storage optimization module.

[0013] Specifically, the preprocessing module uses AI to identify abnormal and irregular data and adopts an outlier detection algorithm, including: Clustering-based anomaly detection: Set two parameters: ε and MinPts, MinPts is the minimum number of samples, ε represents the maximum distance from any given point to its neighboring points, which is used to define the size of the neighborhood. Specifically, in the DBSCAN algorithm, if the distance between two points is less than or equal to ε, the two points are considered to be close to each other.

[0014] For each point, calculate the number of samples in its ε-neighborhood; If a point has at least MinPts points in its ε-neighborhood, then the point is marked as a core point, otherwise it is a non-core point; Among non-core points, if a point does not belong to the ε-neighborhood of any other core point, it is considered an outlier, as follows: ; where Nϵ(p) is the set of points in the ε-neighborhood of point p, D is the entire dataset, and dist(p,q) measures the distance between points p and q.

[0015] Anomaly Detection Based on Sparse Representation: Use sparse coding to learn a low-dimensional representation of the data. Assuming that the data is represented by a sparse linear combination of basis vectors, while the abnormal data cannot be represented, a dictionary E is constructed, where each column is a basis element; For each data point , the optimal sparse code αi is found by solving the following optimization problem: ; Calculate the reconstruction error ( , a high reconstruction error indicates It is abnormal; The optimization problem is the Lasso regression problem, where λ controls the sparsity level, is the L1 norm (sparse regularization term).

[0016] Context-aware time series anomaly detection: Data preparation: Time series data x(t): original time series data; Context information C(t): contains external environmental factors and historical data pattern information; Frequency domain representation X(f): The frequency domain representation obtained by fast Fourier transforming x(t); Frequency Domain Analysis: Apply FFT (Fast Fourier Transformation) to the time series data x(t) to obtain the frequency domain representation X(f): , j represents the imaginary unit, f Indicates frequency; Contextual Information Fusion: The context information C(t) is integrated into the frequency domain representation X(f) to form a context-aware frequency domain representation XC(f): ; Among them, WC(f) is the weight function of context information, which can be adjusted according to the importance of context information; Anomaly Detection: The anomaly detection is performed on the context-aware frequency domain representation XC(f) as follows: Calculate the amplitude: ; Determine abnormal frequency: Set a threshold T. If the amplitude A(f) of a certain frequency f is higher than the background noise level, the moment corresponding to the frequency is considered abnormal: ; Where μA and σA are the mean and standard deviation of the amplitude A(f), respectively, and k is an adjustable parameter; Abnormal point location: For each time point t, check whether its corresponding frequency domain representation XC(f) has a frequency component whose amplitude exceeds the threshold T. If so, the data at time t is considered abnormal.

[0017] The machine learning processing module adopts a method combining support vector machine and adaptive threshold dynamic adjustment, as follows: Model building and training: Decision function: ; Where w is the weight vector, ϕ(x) is the input data mapped to the high-dimensional space, and b is the bias term; Optimization problem: ; Where C is the regularization parameter, which controls the smoothness of the misclassification and decision surface; L is the loss function, which measures the gap between the model prediction value and the true label; The machine learning processing module sends the processed data to the deep learning classification module; The deep learning classification module uses a convolutional neural network deep learning model, which is as follows: The convolutional layer extracts local features: Layer definition: ; in, is the convolution kernel, ∗ represents the convolution operation, is the bias, σ is the activation function, and x is the input; The pooling layer reduces the feature dimension: Max Pooling: ; Fully connected layer decision output: All features are flattened into a vector and fully connected: ; W and b are the weight matrix and bias vector respectively; Output layer classification prediction: The Softmax function transforms this vector z into a probability distribution p, where each element Indicates the probability that the input belongs to the kth category, and uses the Softmax function to make probability predictions for multiple categories: ; is the raw score of the kth class; e is the base of natural logarithms, approximately equal to 2.71828; is the exponential sum of the raw scores of all categories, used for normalization to ensure that the sum of all probabilities is 1; is the output passed from the previous layer; Introducing the attention mechanism to strengthen the convolutional neural network deep learning model's focus on important features: Global average pooling: Perform global average pooling on the feature map of each channel to obtain the feature vector of each channel: ; Where F(i,j,c) is the eigenvalue of the cth channel at position (i,j), H and W are the height and width of the feature map; Global maximum pooling: Perform global maximum pooling on the feature map of each channel to obtain the feature vector of each channel: ; Shared multi-layer perceptron: The results of GAP and GMP are transformed through a shared multi-layer perceptron (MLP) to obtain two intermediate feature vectors: ; ; Where W1 and W2 are the weight matrices of the MLP, and σ is the activation function; Feature fusion: Add the two intermediate feature vectors to get the final channel attention weight: ; Weight application, apply the channel attention weights to the original feature map: ; The deep learning classification module sends the processed data to the real-time data processing module; The real-time data processing module uses real-time anomaly detection algorithm and dynamic threshold adjustment to process real-time data anomalies. The details are as follows: Moving window mean and standard deviation: For each new data point , maintain a fixed-size window W for calculation: ; ; Among them, μt and σt represent the mean and standard deviation within the window respectively; Z-score calculation: For each new data point , calculate its Z-score: ; When ∣ is greater than a certain threshold, then it is considered It is an outlier.

[0018] Dynamic threshold adjustment specifically includes: ; in: is the dynamic threshold at the current moment; is the dynamic threshold at the previous moment; λ is the forget factor, which ranges from 0 to 1 and is used to balance the impact of the historical threshold and the latest Z-score, such as λ=0.95; ∣ ∣ is the absolute value of the Z-score at the current moment.

[0019] Specifically, the data acquisition module: Design a central scheduler to query each data source and pull data at regular intervals; Use Apache Kafka for data transmission, supporting high-throughput information flow management; Each data source has a corresponding API endpoint or SDK integration to standardize the data access format; Preprocessing module: Integrate Python pandas library to handle missing values ​​and format issues; Clustering-based anomaly detection uses the DBSCAN algorithm in the scikit-learn library, and the parameters ε and MinPts are determined in advance through cross-validation; Sparse representation based anomaly detection via Lasso regression, implemented using scikit-learn’s linear_model.Lasso; Context-based anomaly detection: Contextual information fusion: Introducing contextual information into the anomaly detection process to improve the accuracy and interpretability of anomaly detection; Data storage optimization module: Data is partitioned by type and stored in the Apache Hadoop HDFS system to take advantage of its scalability and fault tolerance; Apply data compression algorithms such as Snappy or Zlib to reduce storage space; Machine learning processing module: Use scikit-learn's svm.SVC function to select a suitable kernel function for model training; Use grid search and cross validation to determine the optimal C and kernel parameters; Deep Learning Classification Module: Use TensorFlow to build convolutional layers, pooling layers, and fully connected layers; Design a multi-layer structure to extract features at different levels, using ReLU activation function and maximum pooling technology; The training process includes forward propagation, loss calculation and back propagation, and the Adam optimizer is used to optimize the parameters; Real-time data processing module: Leverage Apache Spark to process real-time data; A sliding window algorithm is used to calculate the moving average and standard deviation of each data point to achieve real-time anomaly detection; Dynamic threshold adjustment adaptively adjusts the abnormality judgment benchmark according to the situation of the previous window; Data quality assurance module: Configure data validation rules and general error checking; Rapid response mechanism to repair or report detected problems; User interface and configuration management module: Developed a web-based console using AngularJS and Bootstrap; Update data and analysis results in real time, and support user-defined queries and report generation; System integration and testing Connect these modules through RESTful APIs, message queues (Kafka), and microservices architecture to ensure data flows seamlessly and securely; The system is deployed on the AWS cloud platform, using its Elastic Load Balancing to ensure load balancing and high availability; Conduct system-level stress testing and user testing to verify functionality, performance, and scalability.

[0020] In summary, the present invention integrates multiple functional modules such as data acquisition, preprocessing, storage optimization, machine learning processing, deep learning classification, real-time data processing, data quality assurance, user interface and configuration management into one system. The integrated design not only improves the efficiency of data processing, but also enhances the overall coordination and ease of use of the system. The adaptive learning mechanism introduced in the machine learning processing module can automatically adjust the model parameters according to the data characteristics, which not only improves the generalization ability of the model, but also enhances the adaptability of the model to different application scenarios, so that the system can maintain high efficiency and accuracy in a constantly changing data environment. By adding context-aware time series anomaly detection, it takes into account the contextual information of time series data and improves the accuracy and reliability of anomaly detection. In the deep learning classification module, in addition to adopting deep learning architectures such as convolutional neural networks, an attention mechanism is also introduced to help the model focus more on important features, thereby improving the accuracy of classification. In this way, the system can better understand complex data structures and make more accurate predictions.

[0021] The system of the present invention integrates multiple modules to achieve a complete process from data collection to analysis and decision support, adapting to different data sources and business needs. By accurately connecting each module, data can flow smoothly through the entire system, ensuring data quality and providing real-time feedback.

[0022] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A big data processing system based on AI cloud computing training, characterized in that: It includes data acquisition module, preprocessing module, data storage optimization module, machine learning processing module, deep learning classification module, real-time data processing module, data quality assurance module, and user interface and configuration management module; The data acquisition module is used to collect data and send the collected data to the preprocessing module and the machine learning processing module respectively. The preprocessing module uses AI to identify abnormal data and irregular data, and then sends the data to the data storage optimization module. The data storage optimization module partitions and stores the data by type. The machine learning processing module uses a method combining support vector machine and adaptive threshold dynamic adjustment to process the data, and then sends the processed data to the deep learning classification module. The deep learning classification module uses a convolutional neural network deep learning model to process the data, and then sends the processed data to the user interface and configuration management module and the real-time data processing module respectively. The user interface and configuration management module update the data and analysis results in real time according to the data processed by the deep learning classification module, and support user-defined queries and report generation. The real-time data processing module uses a real-time anomaly detection algorithm and dynamic threshold adjustment to process real-time data anomalies, and then sends the processed data to the data quality assurance module. The data quality assurance module configures data verification rules, routine error checks, and repairs or reports detected problems according to the data processed by the real-time data processing module and the data stored in the data storage optimization module; The preprocessing module uses AI to identify abnormal data and irregular data, specifically including: Clustering-based anomaly detection: Set two parameters: ε and MinPts, MinPts is the minimum number of samples, and ε represents the maximum distance from any given point to its neighboring points; For each point, calculate the number of samples in its ε-neighborhood; If a point has at least MinPts points in its ε-neighborhood, then the point is marked as a core point, otherwise it is a non-core point; Among non-core points, if a point does not belong to the ε-neighborhood of any other core point, it is considered an outlier, as follows: ； where Nϵ(p) is the set of points in the ε-neighborhood of point p, D is the entire dataset, and dist(p,q) measures the distance between points p and q; The machine learning processing module adopts a method combining support vector machine and adaptive threshold dynamic adjustment, which is as follows: Model building and training: Decision function: ; Where w is the weight vector, ϕ(x) is the input data mapped to the high-dimensional space, and b is the bias term; Optimization problem: ; Where C is the regularization parameter, which controls the smoothness of the misclassification and decision surface; L is the loss function, which measures the gap between the model prediction value and the true label; The deep learning classification module adopts a convolutional neural network deep learning model, which is as follows: The convolutional layer extracts local features: Layer definition: ; in, is the convolution kernel, ∗ represents the convolution operation, is the bias, σ is the activation function, and x is the input; The pooling layer reduces the feature dimension: Max Pooling: ; Fully connected layer decision output: All features are flattened into a vector and fully connected: ; W and b are the weight matrix and bias vector respectively; Output layer classification prediction: The vector z is converted into a probability distribution p through the Softmax function, where each element Indicates the probability that the input belongs to the kth category, and uses the Softmax function to make probability predictions for multiple categories: ; is the raw score of the kth class; e is the base of natural logarithms; is the exponential sum of the raw scores of all categories, used for normalization to ensure that the sum of all probabilities is 1; is the output passed from the previous layer; Introducing the attention mechanism to strengthen the convolutional neural network deep learning model's focus on important features: Global average pooling: Perform global average pooling on the feature map of each channel to obtain the feature vector of each channel: ; Where F(i,j,c) is the eigenvalue of the cth channel at position (i,j), H and W are the height and width of the feature map; Global maximum pooling: Perform global maximum pooling on the feature map of each channel to obtain the feature vector of each channel: ; Shared multi-layer perceptron: The results of GAP and GMP are transformed through a shared multi-layer perceptron to obtain two intermediate feature vectors: ； ； Where W1 and W2 are the weight matrices of the MLP, and σ is the activation function; Feature fusion: Add the two intermediate feature vectors to get the final channel attention weight: ； Weight application: Apply channel attention weights to the original feature map: ； Represents the feature map after applying the channel attention weight; The real-time data processing module uses a real-time anomaly detection algorithm and dynamic threshold adjustment to process real-time data anomaly points, which are specifically as follows: Moving window mean and standard deviation: For each new data point, a fixed-size window W is maintained for calculation: ； ； Among them, μt and σt represent the mean and standard deviation within the window respectively; Z-score calculation: For each new data point , calculate its Z-score: ; When ∣ is greater than a certain threshold, then it is considered It is an outlier.

2. The big data processing system based on AI cloud computing training according to claim 1 is characterized in that: The specific functions of the preprocessing module also include: Anomaly Detection Based on Sparse Representation: Use sparse coding to learn a low-dimensional representation of the data. Assuming that the data is represented by a sparse linear combination of basis vectors, while the abnormal data cannot be represented, a dictionary E is constructed, where each column is a basis element; For each data point , the optimal sparse code αi is found by solving the following optimization problem: ； Calculate the reconstruction error ( , a high reconstruction error indicates It is abnormal; The optimization problem is the Lasso regression problem, where λ controls the sparsity level, is the L1 norm.

3. The big data processing system based on AI cloud computing training according to claim 1 is characterized in that: The specific functions of the preprocessing module also include: Context-aware time series anomaly detection: Data preparation: Time series data x(t): original time series data; Context information C(t): contains external environmental factors and historical data pattern information; Frequency domain representation X(f): The frequency domain representation obtained by performing a fast Fourier transform (FFT) on x(t); Frequency Domain Analysis: Apply FFT transform to the time series data x(t) to obtain the frequency domain representation X(f): , j represents the imaginary unit, f Indicates frequency; Contextual Information Fusion: The context information C(t) is integrated into the frequency domain representation X(f) to form a context-aware frequency domain representation XC(f): ; Among them, WC(f) is the weight function of context information, which can be adjusted according to the importance of context information; Anomaly Detection: The anomaly detection is performed on the context-aware frequency domain representation XC(f) as follows: Calculate the amplitude: ; Determine abnormal frequency: Set a threshold T. If the amplitude A(f) of a certain frequency f is higher than the background noise level, the moment corresponding to the frequency is considered abnormal: ; Where μA and σA are the mean and standard deviation of the amplitude A(f), respectively, and k is an adjustable parameter; Abnormal point location: For each time point t, check whether its corresponding frequency domain representation XC(f) has a frequency component whose amplitude exceeds the threshold T. If so, the data at time t is considered abnormal.

4. The big data processing system based on AI cloud computing training according to claim 1 is characterized in that: Dynamic threshold adjustment specifically includes: ； in, is the dynamic threshold at the current moment, is the dynamic threshold of the previous moment, λ is the forget factor, which ranges from 0 to 1 and is used to balance the influence of the historical threshold and the latest Z-score; It is the absolute value of the Z-score at the current moment.

5. A computer device, comprising a memory, wherein the memory stores program instructions, characterized in that: When the program instructions are executed, the big data processing system based on AI cloud computing training as described in any one of claims 1 to 4 is executed.

Images