Computer data processing method and system

Abstract

The invention discloses a computer data processing method and system, and relates to the technical field of computer data process.The processing method comprises the steps that performance data are obtained, data cleaning is conducted on the performance data, and feature engineering processing is conducted; training the model by using historical performance data according to a selected model algorithm, evaluating the performance of the model, and then performing optimization; wherein the historical performance data is marked as normal; the method is technically characterized in that a machine learning model can automatically learn a normal mode and an abnormal mode of performance data, and abnormality can be recognized more accurately compared with a traditional threshold value method; by continuously learning and updating the model, the system can adapt to the change of the system load and the newly appearing exception type, and the false alarm rate and the missing report rate are reduced; performance abnormity is detected in real time, operation and maintenance personnel are quickly notified, fault discovery and recovery time can be shortened, and system availability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a computer data processing method and system. Background Art

[0002] Computer data processing refers to a series of activities that use computer technology to collect, store, process, analyze and transmit various types of data; it involves the entire life cycle of data, starting from the acquisition of raw data, and organizing and managing data through various means such as database management systems, data warehouses, data mining tools, etc.; in the storage stage, data is safely stored in disks, memory or other storage media for subsequent use; data processing is to clean, convert, integrate and other operations on raw data to meet analysis needs. Data analysis is the core of data processing, which extracts valuable information from data through statistical, machine learning and other methods to provide support for decision-making; finally, the processed data is transmitted to users or systems through the network or other communication means to achieve data sharing and utilization.

[0003] However, the existing methods for processing computer data have the following problems:

[0004] When evaluating performance data, such as CPU usage and memory occupancy, a fixed corresponding threshold is usually set and compared with any indicator in the performance data. If the CPU usage exceeds the corresponding threshold, it means that the system is abnormal. However, this traditional method has low recognition accuracy and cannot adapt to continuously changing systems. If the system load changes, false alarms or missed alarms will occur, and then the operation and maintenance personnel will not be notified quickly, the time for fault discovery will be delayed, and at the same time, the time for system recovery will also be delayed. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the deficiencies of the prior art, the present invention provides a computer data processing method and system, which solves the problems raised in the background technology.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A computer data processing method comprises the following steps:

[0010] Obtain performance data, clean the performance data, and perform feature engineering processing; the performance data includes CPU usage, memory usage, disk read and write speed, network throughput, and timestamp and server identification information;

[0011] Based on the selected model algorithm, the model is trained using historical performance data, and the model performance is evaluated and then optimized; the historical performance data is marked as normal;

[0012] Deploy the trained model to the performance monitoring system and set a threshold to determine whether the performance data is abnormal. If the performance data exceeds the threshold, it will be marked as abnormal and trigger an alarm signal.

[0013] Combine system logs, configuration information, and domain knowledge to analyze abnormal performance data and obtain the cause of the abnormality.

[0014] Furthermore, the performance data is obtained by using a monitoring tool; wherein the monitoring tool includes any one of Prometheus and Zabbix;

[0015] CPU usage: indicates the percentage of CPU used within a set time interval;

[0016] Memory usage: indicates the percentage of memory occupied within a set time interval;

[0017] Disk read and write speed: indicates the disk read and write speed within a set time interval;

[0018] Network throughput: refers to the amount of data transmitted over the network within a set time interval;

[0019] Timestamp: indicates the time point when the recorded performance data is collected;

[0020] Server identification information: including server ID, IP address, and operating system type.

[0021] Furthermore, data cleaning means removing invalid or erroneous data, handling missing values, and normalizing performance data;

[0022] The process of feature engineering is as follows:

[0023] Constructing basic features: constructing basic features, including the moving average, standard deviation, rate of change, and peak value of the indicator; where the indicator represents any type of data in the performance data;

[0024] Construct correlation features: construct correlation features between different indicators;

[0025] Feature selection and dimensionality reduction: Statistical methods are used to perform feature selection and dimensionality reduction; wherein the statistical methods include any one of correlation analysis and variance analysis.

[0026] Further, the selected model algorithm includes any one of a time series analysis algorithm and an anomaly detection algorithm;

[0027] Build a rule engine. For performance data, if the focus is on time series patterns or long-term dependencies, select a time series analysis algorithm, and the time series analysis algorithm is LSTM or GRU; if the focus is on the local density and clustering structure of the data, select an anomaly detection algorithm, and the anomaly detection algorithm is Isolation Forest or LOF.

[0028] Further, the specific steps for training the model are as follows:

[0029] Data preparation: Arrange the historical performance data in chronological order and divide it into a training set and a validation set according to a ratio of 7:3; among them, the training set is used to train the model, and the validation set is used to evaluate the model performance.

[0030] Model initialization: Initialize the model parameters according to the selected algorithm; among them, in LSTM or GRU, the model parameters include the weights and biases of the network layers; in Isolation Forest or LOF, the model parameters include the number of trees and the maximum depth.

[0031] Model training: Iteratively update the model parameters using the training set data.

[0032] Model saving: After training is completed, save the model parameters.

[0033] Further, after evaluating the model performance, the optimization is as follows: Use the holdout method to evaluate the model performance, obtain the performance evaluation metrics, and perform weighted calculations based on the performance evaluation metrics to obtain the comprehensive performance valuation. When the comprehensive performance valuation is lower than the preset value, trigger the model optimization strategy; otherwise, there is no response; among them, the performance evaluation metrics include accuracy, recall, and F1-score.

[0034] The model optimization strategy is: Hyperparameter tuning, using a tuning tool to find the best combination of model parameters.

[0035] Among them, the tuning tools include Hyperopt and Optuna.

[0036] Further, monitor the load of the performance monitoring system in real time. When the load exceeds the system-calibrated load, trigger the dynamic adjustment mechanism and construct a threshold calculation formula as follows:

[0037]

[0038] In the formula, T 1 represents the adjusted threshold, T 0 represents the current threshold, k represents the adjustment coefficient, and the value range is: 0 < k < 1; L represents the current load, and Le represents the system-calibrated load.

[0039] Furthermore, after the alarm signal is triggered, the alarm information is sent to the operation and maintenance end; wherein the alarm information includes the time when the abnormality occurs, the server identifier, the abnormality indicator and its value.

[0040] Furthermore, the process of obtaining the cause of the exception is as follows:

[0041] Log analysis: Use log analysis technology to analyze the logs before and after the exception occurs, search the logs by keywords, thread names, and class names, and locate the log entries related to the exception; analyze the log content to identify the exception pattern, error type, and resource contention;

[0042] Configuration information comparison: compare the configuration information when the anomaly occurs with the configuration information in the normal state, identify the difference in configuration information, and analyze whether the difference is related to the performance anomaly; the technologies used to analyze the difference include text comparison technology and data mining technology;

[0043] Application of domain knowledge: Comprehensively analyze abnormal data and log information in combination with domain knowledge; domain knowledge includes system architecture, component relationships, and common failure modes, and use the domain knowledge base to determine the root cause of the abnormality.

[0044] Text matching technology: used to compare the text information in the configuration file item by item.

[0045] Data mining technology: used to extract key information from configuration information and perform correlation analysis.

[0046] A computer data processing system, the system comprising:

[0047] The data collection and processing module obtains performance data, cleans the performance data, and performs feature engineering processing; the performance data includes CPU usage, memory occupancy, disk read and write speed, network throughput, timestamp, and server identification information;

[0048] The model selection training module uses historical performance data to train the model and then optimizes the model after evaluating its performance. The historical performance data is marked as normal.

[0049] The real-time monitoring feedback module deploys the trained model to the performance monitoring system and sets a threshold to determine whether the performance data is abnormal. Performance data that exceeds the threshold is marked as abnormal and triggers an alarm signal.

[0050] The result analysis module combines system logs, configuration information and domain knowledge to analyze abnormal performance data and obtain the causes of the abnormalities.

[0051] (III) Beneficial effects

[0052] The present invention provides a computer data processing method and system, which have the following beneficial effects:

[0053] (1) This solution first ensures the accuracy, completeness and consistency of the data through comprehensive data collection and preprocessing, laying a solid foundation for subsequent analysis; the introduction of feature engineering effectively extracts key information from performance data and enhances the expressiveness of the model; in the model selection and training stage, based on the time series characteristics of server performance data and the needs of anomaly detection, a suitable algorithm is selected, and the model learns the normal performance mode by training historical data; in the model evaluation and tuning stage, a variety of evaluation methods and performance indicators are used to systematically optimize the model parameters and hyperparameters, significantly improving the accuracy and efficiency of anomaly detection;

[0054] Overall, this technical solution not only achieves real-time and accurate detection of server performance anomalies, but also improves the generalization ability and stability of the model through continuous iterative optimization, provides strong technical support for server operation and maintenance, effectively reduces system risks and operation and maintenance costs caused by performance anomalies, and improves service quality and user experience.

[0055] (2) This solution achieves all-round monitoring and anomaly detection of server performance, significantly improving operation and maintenance efficiency and system stability. By deploying models in real time, the system can efficiently identify and alarm performance anomalies, ensuring that operation and maintenance personnel can respond and handle them quickly. At the same time, the dynamic threshold adjustment mechanism automatically optimizes the sensitivity of anomaly detection according to the system load, improving the accuracy of detection. The root cause analysis process is automated, combining log analysis, configuration information comparison and domain knowledge to quickly locate the root cause of anomalies, reducing manual troubleshooting time and improving problem solving efficiency. In addition, the continuous learning and updating mechanism ensures that the model can adapt to system changes and capture new anomaly patterns in a timely manner. The entire solution integrates advanced algorithms and tools, such as LSTM, ELK Stack, etc., providing powerful data analysis and visualization capabilities.

[0056] In summary, this technical solution not only realizes the intelligence and automation of performance monitoring, but also significantly improves the initiative and response speed of system operation and maintenance, provides strong guarantees for the stable operation of the server, and greatly reduces the risk of business interruption caused by performance abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The figure is a flowchart of the overall steps of a computer data processing method in the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Embodiment 1:

[0060] See also Figure 1 This embodiment provides a computer data processing method, which includes the following steps:

[0061] S1. Data collection and preprocessing:

[0062] Collect data: Obtain performance data through monitoring tools (such as Prometheus, Zabbix) or directly from the server operating system, including CPU usage, memory usage, disk read and write speed, network throughput, etc., as well as timestamps and server identification information;

[0063] Among them, in the server performance monitoring system, data collection is the first and most important step. We need to obtain data reflecting the server operation status from various sources;

[0064] Monitoring Tools:

[0065] Prometheus: This is an open source system monitoring and alarm toolkit, which is widely used in cloud native environments. It can collect time series data of various indicators, such as CPU usage, memory usage, etc., and provide powerful query and alarm functions; Zabbix: This is an enterprise-level open source solution for monitoring various network devices, servers, applications, etc.; Zabbix can collect detailed performance data and support graphical monitoring interface and alarm mechanism; Through these monitoring tools, we can configure corresponding monitoring items and triggers, collect server performance data regularly (such as every minute, every hour), and store it in the database or send it to the specified data processing system;

[0066] Directly from the server operating system:

[0067] In some cases, we may need to obtain performance data directly from the server's operating system level. This usually involves reading system files (such as files in the / proc directory in Linux systems) or using system commands (such as top, vmstat, etc.) to obtain real-time performance information; this method may require writing scripts or using specific tools to execute these commands regularly and parse the results into a data format that can be used for subsequent analysis;

[0068] The data collected shall include but not be limited to:

[0069] CPU usage: indicates the percentage of CPU used within a set time interval;

[0070] Memory usage: indicates the percentage or specific amount of memory (RAM) occupied within a set time interval;

[0071] Disk read and write speed: indicates the read and write speed of the disk within a set time interval, usually in bytes / second or IOPS (input / output operations per second);

[0072] Network throughput: refers to the amount of data transmitted over a network within a set time interval, usually measured in bytes per second;

[0073] Timestamp: records the time point of data collection for subsequent time series analysis;

[0074] Server identification information: such as server ID, IP address, operating system type, etc., used to distinguish data from different servers;

[0075] Data cleaning: remove invalid or erroneous data (such as negative values, values ​​outside a reasonable range), handle missing values ​​(such as interpolation filling), and normalize or standardize the data to eliminate the impact of different indicator dimensions;

[0076] Among them, data cleaning is a key step in ensuring data quality, which involves identifying and correcting errors, anomalies or inconsistencies in the data;

[0077] Remove invalid or erroneous data:

[0078] Check whether the data contains negative values ​​or values ​​that are out of a reasonable range (such as CPU usage exceeding 100%). These values ​​may be caused by errors in monitoring tools, system failures, or data transmission errors; for these invalid or erroneous data points, you can choose to delete, replace with default values, or use other methods to handle them;

[0079] Handling missing values:

[0080] During the data collection process, data may be missing at certain time points due to network delays, monitoring tool failures, or server downtime. For missing values, you can use interpolation (such as linear interpolation, polynomial interpolation) or fill in the missing values ​​based on the average of the previous and next data points. If there are many missing values ​​or they are missing continuously, you may need to consider using more complex data completion methods.

[0081] Data normalization or standardization:

[0082] Different indicators may have different dimensions and value ranges (e.g., CPU usage is between 0-100%, while network throughput may be in MB / s or GB / s). In order to eliminate the impact of these differences on subsequent analysis, the data needs to be normalized or standardized; normalization usually scales the data to a specific range (such as 0-1), while standardization converts the data into a distribution with a mean of 0 and a standard deviation of 1. Which method to choose depends on the specific application scenario and algorithm requirements;

[0083] Feature engineering: Based on the experience of performance monitoring, construct features such as the moving average, standard deviation, rate of change, peak value, etc. of indicators, as well as the correlation features between different indicators (such as the ratio of CPU usage to memory usage);

[0084] Among them, feature engineering is one of the key steps in machine learning, which involves extracting useful features from raw data to improve the performance of the model.

[0085] Construction basic features:

[0086] Based on the experience and domain knowledge of performance monitoring, some basic features can be constructed, such as the moving average, standard deviation, rate of change, peak value, etc. of the indicator; the moving average can reflect the average level of the indicator over a period of time; the standard deviation can reflect the degree of fluctuation of the indicator; the rate of change can reflect the changing trend of the indicator; and the peak value can reflect the maximum value of the indicator at a specific time point.

[0087] Construct associative features:

[0088] In addition to basic features, we can also construct correlation features between different indicators. For example, the ratio of CPU usage to memory occupancy can reflect the load relationship between CPU and memory; the ratio of network throughput to disk read and write speed can reflect the efficiency relationship between data transmission and storage. These correlation features help capture the interactions and dependencies between different indicators and provide richer information for the model.

[0089] Feature selection and dimensionality reduction:

[0090] After constructing a large number of features, feature selection and dimensionality reduction may be required; feature selection aims to select the features that have the greatest impact on model performance, while dimensionality reduction aims to reduce the number and dimensions of features to reduce model complexity and computational cost; feature selection and dimensionality reduction can be performed using statistical methods (such as correlation analysis, variance analysis) or machine learning algorithms (such as random forest, Lasso regression).

[0091] S2. Model selection and training:

[0092] Select an algorithm: Considering the time series characteristics of server performance data and the need for anomaly detection, you can choose a time series analysis algorithm (such as LSTM, GRU) or an anomaly detection algorithm (such as Isolation Forest, LOF);

[0093] Among them, when selecting an algorithm, we need to consider the characteristics of server performance data and the specific needs of anomaly detection; server performance data usually has time series characteristics, that is, the data is arranged in time order, and there may be some dependencies between adjacent data points; in addition, our goal is to detect anomalies in performance data, that is, those data points that are significantly different from the normal data distribution;

[0094] Time Series Analysis Algorithms:

[0095] LSTM (Long Short-Term Memory): LSTM is a special recurrent neural network (RNN) that can capture dependencies in long time series. It solves the gradient vanishing and gradient exploding problems in traditional RNNs by introducing input gates, forget gates, and output gates, and is suitable for processing time series data with long-term dependencies. In server performance anomaly detection, LSTM can learn the time series patterns of normal performance data and use them to predict and detect anomalies. GRU (Gated Recurrent Unit): GRU is another improved RNN that is similar to LSTM but has a simpler structure. It also controls the flow of information through a gating mechanism to capture dependencies in time series data. GRU is generally comparable to LSTM in performance, but has a lower computational cost, so it may be more applicable in some cases.

[0096] Anomaly Detection Algorithms:

[0097] Isolation Forest: Isolation Forest is a tree-based anomaly detection algorithm that constructs multiple binary trees (called "isolation trees") by randomly selecting features and split values, thereby "isolating" data points into the leaf nodes of the tree; abnormal data points are usually more easily isolated into shallower leaf nodes, so the "isolation depth" of the data point can be calculated to determine whether it is an anomaly; Isolation Forest is suitable for processing high-dimensional data and large-scale data sets; LOF (Local Outlier Factor): LOF is a density-based anomaly detection algorithm that detects anomalies by calculating the local density ratio between a data point and its neighbors; if the local density of a data point is significantly lower than the average density of its neighbors, it is considered an anomaly; LOF is suitable for processing data sets with obvious clustering structures.

[0098] When choosing an algorithm, we need to consider the characteristics of the data (such as time series, dimension, scale, etc.) and the specific requirements of anomaly detection (such as real-time, accuracy, explainability, etc.). For server performance data, if you are more concerned about time series patterns and long-term dependencies, you can choose LSTM or GRU; if you are more concerned about the local density and clustering structure of the data, you can choose Isolation Forest or LOF.

[0099] Training model: Use historical performance data (marked as normal) to train the model and let the model learn the distribution and change trend of normal performance data;

[0100] After selecting a suitable algorithm, we need to use historical performance data to train the model; historical performance data is usually marked as normal because our goal is to let the model learn the distribution and change trend of normal data so as to identify abnormal data in subsequent detection;

[0101] The specific steps of training the model include:

[0102] Data preparation: Arrange the historical performance data in chronological order and divide it into a training set and a validation set (or test set); the training set is used to train the model, and the validation set (or test set) is used to evaluate the model performance;

[0103] Model initialization: Initialize the model parameters according to the selected algorithm; for example, in LSTM or GRU, the weights and biases of the network layers need to be initialized; in Isolation Forest or LOF, the number of trees, maximum depth and other parameters need to be initialized;

[0104] Model training: Iteratively update model parameters using training set data; for neural network models such as LSTM or GRU, backpropagation algorithm and gradient descent (or its variants) are usually used to optimize the loss function; for tree models such as isolation forest or LOF, the model is trained by building trees and calculating local density;

[0105] Model saving: After training is completed, the model parameters are saved for use in subsequent tests.

[0106] Model evaluation and tuning: Evaluate model performance through methods such as holdout and time series cross-validation, and adjust model parameters and hyperparameters to optimize anomaly detection results;

[0107] After training the model, we need to evaluate its performance and perform optimization. The purpose of model evaluation is to measure the performance of the model on unknown data, while optimization is to improve model performance by adjusting model parameters and hyperparameters.

[0108] Model evaluation method:

[0109] Holdout method: Divide the dataset into training set and test set, use the training set to train the model, and evaluate the model performance on the test set; this method is simple and direct, but may be affected by the way the data is divided; Time series cross-validation: For time series data, time series cross-validation can be used to evaluate model performance; this method divides the dataset into multiple time windows, each of which contains a portion of training data and a portion of test data; then, train the model on each time window and evaluate its performance, and finally calculate the average performance index. This method can better evaluate the generalization ability of the model on time series data.

[0110] Performance evaluation indicators:

[0111] Accuracy: For classification problems (such as classifying performance data into normal and abnormal categories), accuracy can be used to measure the correctness of model predictions; Recall and precision: In anomaly detection, we are more concerned about the detection of abnormal data. Recall indicates the proportion of abnormal data that are correctly detected to all abnormal data, while precision indicates the proportion of data detected as abnormal that are actually abnormal; F1 score: The F1 score is the harmonic mean of recall and precision, which is used to comprehensively evaluate model performance; ROC curve and AUC value: For anomaly detection algorithms that require setting thresholds (such as isolation forests), ROC curves and AUC values ​​can be used to evaluate the performance of the model at different thresholds; The ROC curve plots the relationship between the true positive rate (recall rate) and the false positive rate, and the AUC value is the area under the ROC curve, which is used to quantify the classification ability of the model.

[0112] Model tuning:

[0113] Parameter tuning: Improve model performance by adjusting model parameters (such as learning rate, number of layers, number of neurons in neural networks, etc.). You can use grid search, random search, or Bayesian optimization to find the best parameter combination; Hyperparameter tuning: Hyperparameters are parameters set before model training (such as the number of trees in an isolated forest, maximum depth, etc.). They are not directly involved in model training but affect model performance. You can find the best hyperparameter combination through empirical selection, cross-validation, or automated parameter tuning tools (such as Hyperopt, Optuna, etc.);

[0114] Through continuous evaluation, tuning, and iteration, we can gradually improve the accuracy and efficiency of the model in detecting server performance anomalies. After adopting the above technical solutions, the technical effect is significantly improved;

[0115] First, through comprehensive data collection and preprocessing, we ensured the accuracy, completeness and consistency of the data, laying a solid foundation for subsequent analysis. The introduction of feature engineering effectively extracted key information from performance data and enhanced the expressiveness of the model. In the model selection and training phase, we selected suitable algorithms based on the time series characteristics of server performance data and the needs of anomaly detection, and trained the model with historical data to learn the normal performance mode. In the model evaluation and tuning phase, we used a variety of evaluation methods and performance indicators to systematically optimize model parameters and hyperparameters, significantly improving the accuracy and efficiency of anomaly detection.

[0116] Overall, this technical solution not only realizes real-time and accurate detection of server performance anomalies, but also improves the generalization ability and stability of the model through continuous iterative optimization, providing strong technical support for server operation and maintenance, effectively reducing system risks and operation and maintenance costs caused by performance anomalies, and improving service quality and user experience.

[0117] S3, Real-time detection and feedback:

[0118] Deploy the model: Deploy the trained model to the performance monitoring system to detect anomalies in real-time performance data.

[0119] Deploying the trained model to the performance monitoring system is a key step in achieving real-time performance data anomaly detection. The following are some specific deployment considerations:

[0120] Model integration: Integrate trained models (such as LSTM, GRU, Isolation Forest or LOF, etc.) into the performance monitoring system; this may require the development of a model service interface that can receive real-time performance data and return the anomaly detection results of the model; Resource allocation: According to the computing requirements of the model and the load of the performance monitoring system, reasonably allocate computing resources (such as CPU, memory, GPU, etc.) to the model service; ensure that the model can complete the anomaly detection task within the specified time without affecting other functions of the monitoring system; Security and reliability: Ensure the security of model deployment to prevent unauthorized access and malicious attacks; at the same time, take necessary redundancy and backup measures to improve the reliability of model services;

[0121] Abnormality determination: According to the abnormality score or label output by the model, set a threshold to determine whether the data is abnormal. For performance data exceeding the threshold, it is marked as abnormal and an alarm is triggered;

[0122] Among them, during the real-time detection process, the system needs to determine whether the data is abnormal based on the anomaly score or label output by the model; the following are some specific judgment methods:

[0123] Threshold setting: Set a reasonable threshold according to the output characteristics of the model (such as the distribution range of anomaly scores) and actual requirements (such as the sensitivity and specificity of anomaly detection); when the anomaly score output by the model exceeds this threshold, the data is considered abnormal; Dynamic adjustment: The threshold can be dynamically adjusted according to the operating conditions of the system and the effect of anomaly detection; for example, when the system load is high or the anomaly pattern changes, the threshold can be appropriately reduced to improve the sensitivity of anomaly detection; Multiple determination: In some cases, a multiple determination strategy can be adopted to improve the accuracy of anomaly detection; for example, the anomaly detection results of multiple models can be combined for comprehensive determination, or a rule-based determination method can be used to further screen abnormal data;

[0124] Among them, the process of setting the threshold in this embodiment is as follows:

[0125] Monitor the load of the performance monitoring system in real time. When the load exceeds the calibrated load of the system, trigger the dynamic adjustment mechanism and construct the threshold calculation formula as follows:

[0126]

[0127] In the formula, T 1 represents the adjusted threshold, T 0 represents the current threshold, k represents the adjustment coefficient, and the value range is: 0 < k < 1; L represents the current load, and Le represents the calibrated load of the system;

[0128] The logical explanation is as follows:

[0129] First, calculate the difference between the current load L and the calibrated load Le of the system. This difference reflects the degree to which the system load exceeds the calibrated load of the system, and the calibrated load of the system is set in advance according to the actual situation; divide the difference by the calibrated load of the system to obtain a proportional value, which represents the relative size of the load deviation from the calibrated load of the system; multiply by the adjustment coefficient k (0 < k < 1) to control the adjustment amplitude of the threshold; subtract an adjustment amount from the current threshold T to obtain the adjusted threshold;

[0130] When L > Le (that is, the system load is higher than the calibrated load of the system), (L - Le) / Le is positive. Therefore, the adjusted threshold will be less than the current threshold, reducing the threshold and improving the sensitivity of anomaly detection; when L < Le (that is, the system load is lower than the calibrated load of the system), although the formula still applies, usually in this case, it may not be necessary to reduce the sensitivity. Therefore, unnecessary adjustments can be avoided by setting the value of k or introducing other logics;

[0131] Alarm and response: When an abnormality is detected, the system immediately sends an alarm message to the operation and maintenance personnel, including the time when the abnormality occurred, server ID, abnormal indicators and their values, so that the problem can be quickly located and measures can be taken;

[0132] When an abnormality is detected, the system needs to immediately send an alarm message to the operation and maintenance personnel so that the problem can be quickly located and measures can be taken; the following are some specific alarm and response measures:

[0133] Alarm information: Alarm information should include key information such as the time when the exception occurred, server identification, abnormal indicators and their values, so that operation and maintenance personnel can quickly understand the abnormal situation and locate the source of the problem; Alarm channel: According to the work habits and actual situation of the operation and maintenance personnel, select appropriate alarm channels (such as email, SMS, instant messaging tools, etc.) to send alarm information. Ensure that the alarm information can be conveyed to relevant personnel in a timely and accurate manner; Response process: Develop a clear abnormal response process, including abnormal confirmation, problem troubleshooting, fault repair and other steps. Ensure that the operation and maintenance personnel can handle abnormal events efficiently according to the process and restore the normal operation of the system as soon as possible;

[0134] Continuous learning and updating: As new performance data is generated, the model is updated regularly to adapt to changes in system load and emerging abnormal patterns;

[0135] As new performance data is generated, the system needs to regularly update the model to adapt to changes in system load and emerging abnormal patterns; the following are some specific methods for continuous learning and updating:

[0136] Data collection: Continuously collect real-time performance data and use it to update and train the model. Ensure that the new data can reflect the latest operating status and abnormal patterns of the system; Model retraining: Regularly retrain the model with newly collected data. This can be achieved through methods such as incremental learning or online learning to improve the adaptability and accuracy of the model; Effect evaluation: After the model is updated, use the validation set or test set to evaluate the model to ensure that its performance meets actual needs. If it is found that the model performance has degraded or cannot meet the requirements, it is necessary to further adjust the model parameters or adopt a new algorithm for training; Automated update: In order to achieve continuous learning and updating of the model, you can consider automating the model training, evaluation and update process. For example, you can develop an automated script or pipeline to perform these tasks regularly and monitor the performance changes of the model to trigger necessary update operations;

[0137] Through the above steps, real-time monitoring and anomaly detection of server performance can be achieved, providing timely and accurate alarm information to operation and maintenance personnel, and helping the system adapt to changes in load and abnormal patterns;

[0138] S4. Root cause analysis of results:

[0139] Combine system logs, configuration information, and domain knowledge to conduct in-depth analysis of abnormal data and identify the root causes of performance anomalies (such as resource contention, configuration errors, hardware failures, etc.);

[0140] Root Cause Analysis:

[0141] Log analysis: Use log analysis techniques (such as log search, log parsing, log pattern recognition, etc.) to conduct in-depth analysis of logs before and after the exception occurs; search logs by keywords, thread names, class names, etc. to quickly locate log entries related to the exception; analyze log content to identify possible exception patterns, error types, and resource contention;

[0142] Configuration information comparison: Compare the configuration information when the anomaly occurs with the configuration information in the normal state; identify the differences and changes in the configuration information, and analyze whether these differences are related to the performance anomaly;

[0143] Application of domain knowledge: Combine domain knowledge (such as system architecture, component relationships, common failure modes, etc.) to conduct comprehensive analysis of abnormal data and log information; use empirical rules and domain knowledge base to assist in determining the root cause of the abnormality.

[0144] Process Automation:

[0145] Use automated scripts or workflow engines (such as Airflow, Jenkins, etc.) to connect steps such as anomaly detection, log analysis, configuration information comparison, and domain knowledge application to form an automated root cause analysis process; set up scheduled tasks or event trigger mechanisms to ensure that the root cause analysis process can be automatically executed regularly or when anomalies occur; Result display and alarm: Display the results of the root cause analysis in a visual way, such as dashboards and reports; When the root cause analysis is completed, automatically send alarm information to relevant personnel, including the results of the root cause analysis and recommended solutions;

[0146] It can also include continuous learning and optimization:

[0147] Collect feedback and results from the root cause analysis process to continuously optimize the anomaly detection model and root cause analysis process; continuously learn and accumulate domain knowledge to improve the accuracy and efficiency of root cause analysis.

[0148] Technology selection and tools

[0149] Log analysis tools: such as Splunk, ELK Stack (Elasticsearch, Logstash, Kibana), etc.

[0150] Anomaly detection algorithms: such as LSTM, GRU, Isolation Forest, LOF, etc.;

[0151] Workflow engines: such as Airflow, Jenkins, etc.

[0152] Data visualization tools: such as Tableau, Power BI, etc.

[0153] Through the above solution, root cause analysis can be automated to find the root cause of performance anomalies without excessive human involvement; this can not only improve the efficiency and accuracy of root cause analysis, but also reduce the workload of operation and maintenance personnel.

[0154] After adopting the above technical solutions, all-round monitoring and anomaly detection of server performance are achieved, which significantly improves operation and maintenance efficiency and system stability. By deploying models in real time, the system can efficiently identify and alarm performance anomalies, ensuring that operation and maintenance personnel can respond and handle them quickly. At the same time, the dynamic threshold adjustment mechanism automatically optimizes the sensitivity of anomaly detection according to the system load, improving the accuracy of detection. The root cause analysis process is automated, combining log analysis, configuration information comparison and domain knowledge to quickly locate the root cause of anomalies, reducing manual troubleshooting time and improving problem solving efficiency. In addition, the continuous learning and updating mechanism ensures that the model can adapt to system changes and capture new anomaly patterns in a timely manner. The entire solution integrates advanced algorithms and tools, such as LSTM, ELK Stack, etc., providing powerful data analysis and visualization capabilities.

[0155] In summary, this technical solution not only realizes the intelligence and automation of performance monitoring, but also significantly improves the initiative and response speed of system operation and maintenance, provides strong guarantees for the stable operation of the server, and greatly reduces the risk of business interruption caused by performance abnormalities.

[0156] Example:

[0157] Suppose in a cloud computing environment, a server is responsible for processing user requests and storing data. By deploying a server performance anomaly detection system based on machine learning, the system can automatically analyze the performance data of the server. One day, the system detects that the CPU usage of the server suddenly soared, far exceeding the historical average, and the memory usage rate continued to rise. The system immediately marked these performance data as abnormal and sent an alarm message to the operation and maintenance personnel. The operation and maintenance personnel checked the system logs and monitoring charts and found that it was caused by a memory leak in a certain application. They quickly located and fixed the problem, avoiding the risk of system crash and service interruption.

[0158] Specifically, the effects achieved by the above method are as follows:

[0159] Improve anomaly detection accuracy: Machine learning models can automatically learn normal and abnormal patterns of performance data and identify anomalies more accurately than traditional threshold methods. Reduce false positive and false negative rates: By continuously learning and updating models, the system can adapt to changes in system load and emerging anomaly types, reducing false positive and false negative rates. Shorten fault recovery time: Real-time detection of performance anomalies and rapid notification of operation and maintenance personnel help shorten fault discovery and recovery time and improve system availability. Provide performance optimization suggestions: Through in-depth analysis of abnormal data, the system can provide performance optimization suggestions to operation and maintenance personnel, such as adjusting resource configuration, optimizing application code, etc., to improve the overall performance of the system.

[0160] Embodiment 2:

[0161] Based on Example 1, this embodiment further provides a computer data processing system, which includes:

[0162] The data collection and processing module obtains performance data, cleans the performance data, and performs feature engineering processing; the performance data includes CPU usage, memory occupancy, disk read and write speed, network throughput, timestamp, and server identification information;

[0163] The model selection training module uses historical performance data to train the model and then optimizes the model after evaluating its performance. The historical performance data is marked as normal.

[0164] The real-time monitoring feedback module deploys the trained model to the performance monitoring system and sets a threshold to determine whether the performance data is abnormal. Performance data that exceeds the threshold is marked as abnormal and triggers an alarm signal.

[0165] The result analysis module combines system logs, configuration information and domain knowledge to analyze abnormal performance data and obtain the causes of the abnormalities.

[0166] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A computer data processing method, characterized in that: The steps include: Obtain performance data, clean the performance data, and perform feature engineering processing; the performance data includes CPU usage, memory usage, disk read and write speed, network throughput, and timestamp and server identification information; Based on the selected model algorithm, the model is trained using historical performance data, and the model performance is evaluated and then optimized; the historical performance data is marked as normal; Deploy the trained model to the performance monitoring system and set a threshold to determine whether the performance data is abnormal. If the performance data exceeds the threshold, it will be marked as abnormal and trigger an alarm signal. Combine system logs, configuration information, and domain knowledge to analyze abnormal performance data and obtain the cause of the abnormality.

2. A computer data processing method according to claim 1, characterized in that: The performance data is obtained by using a monitoring tool; the monitoring tool includes any one of Prometheus and Zabbix; CPU usage: indicates the percentage of CPU used within a set time interval; Memory usage: indicates the percentage of memory occupied within a set time interval; Disk read and write speed: indicates the disk read and write speed within a set time interval; Network throughput: refers to the amount of data transmitted over the network within a set time interval; Timestamp: indicates the time point when the recorded performance data is collected; Server identification information: including server ID, IP address, and operating system type.

3. A computer data processing method according to claim 1, characterized in that: Data cleaning means removing invalid or erroneous data, handling missing values, and normalizing performance data; The process of feature engineering is as follows: Constructing basic features: constructing basic features, including the moving average, standard deviation, rate of change, and peak value of the indicator; where the indicator represents any type of data in the performance data; Construct correlation features: construct correlation features between different indicators; Feature selection and dimensionality reduction: Statistical methods are used to perform feature selection and dimensionality reduction; wherein the statistical methods include any one of correlation analysis and variance analysis.

4. A computer data processing method according to claim 1, characterized in that: The selected model algorithm includes any one of a time series analysis algorithm and an anomaly detection algorithm; Build a rule engine. For performance data, if the focus is on time series patterns or long-term dependencies, select a time series analysis algorithm, and the time series analysis algorithm is LSTM or GRU; if the focus is on the local density and clustering structure of the data, select an anomaly detection algorithm, and the anomaly detection algorithm is isolation forest or LOF.

5. A computer data processing method according to claim 1 or 4, characterized in that: The specific steps for training the model are as follows: Data preparation: Arrange the historical performance data in chronological order and divide it into a training set and a validation set in a ratio of 7:

3. The training set is used to train the model, and the validation set is used to evaluate the model performance. Model initialization: Initialize the model parameters according to the selected algorithm; in LSTM or GRU, the model parameters include the weights and biases of the network layers; in Isolation Forest or LOF, the model parameters include the number of trees and the maximum depth; Model training: iteratively update model parameters using training set data; Model saving: After training is completed, the model parameters are saved.

6. A computer data processing method according to claim 5, characterized in that: After evaluating the model performance, the following is for tuning: Use the holdout method to evaluate the model performance, obtain the performance evaluation metrics, and perform weighted calculations based on the performance evaluation metrics to obtain the comprehensive performance valuation. When the comprehensive performance valuation is lower than the preset value, trigger the model tuning strategy; Conversely, there is no response; among them, the performance evaluation metrics include accuracy, recall, and F1 score; The model tuning strategy is: Hyperparameter tuning, using a tuning tool to find the best combination of model parameters; Among them, the tuning tools include Hyperopt and Optuna.

7. A computer data processing method according to claim 1, characterized in that: The process of setting the threshold is: Real-time monitor the load of the performance monitoring system. When the load exceeds the system-calibrated load, trigger the dynamic adjustment mechanism and construct the threshold calculation formula as follows: In the formula, T1 represents the adjusted threshold, T0 represents the current threshold, k represents the adjustment coefficient, and the value range is: 0 < k < 1; L represents the current load, and Le represents the system-calibrated load.

8. A computer data processing method according to claim 1, characterized in that: After triggering the alarm signal, send the alarm information to the operation and maintenance side; among them, the alarm information includes the time of the anomaly occurrence, the server identifier, the anomaly metrics and their values.

9. A computer data processing method according to claim 1, characterized in that: The process of obtaining the cause of the anomaly is as follows: Log analysis: Use log analysis technology to analyze the logs before and after the anomaly occurrence, search for logs through keywords, thread names, and class names, and locate the log entries related to the anomaly; analyze the log content to identify anomaly patterns, error types, and resource contention situations; Configuration information comparison: Compare the configuration information at the time of the anomaly with the configuration information in the normal state, identify the differences in the configuration information, and analyze whether the differences are related to the performance anomaly; among them, the technologies used to analyze the differences include text comparison technology and data mining technology; Domain knowledge application: Combine domain knowledge to comprehensively analyze the anomaly data and log information; among them, the domain knowledge includes system architecture, component relationships, and common fault patterns, and use the domain knowledge base to judge the root cause of the anomaly.

10. A computer data processing system, used to implement the processing method according to any one of claims 1 to 9, characterized in that: The system includes: Data collection and processing module, obtain performance data, perform data cleaning on the performance data, and perform feature engineering processing; among them, the performance data includes CPU usage rate, memory occupancy rate, disk read and write speed, network throughput, as well as timestamp and server identifier information; Model selection and training module, use historical performance data to train the model, and perform tuning after evaluating the model performance; among them, the historical performance data is marked as normal; Real-time monitoring and feedback module, deploy the trained model to the performance monitoring system, set the threshold to judge whether the performance data is abnormal; among them, the performance data exceeding the threshold is marked as abnormal and triggers the alarm signal; Result analysis module, combine system logs, configuration information, and domain knowledge to analyze the abnormal performance data and obtain the cause of the anomaly.

Images