Intelligent data monitoring method and system for server

By using machine learning models in the server monitoring system to analyze historical and real-time data, dynamically adjust monitoring frequency and alarm, the complex data management problems in large-scale environments are solved, resource efficiency and abnormal detection capabilities are improved, and operation and maintenance are reduced Cost enhances system stability.

CN118939508APending Publication Date: 2024-11-12FUXI SEMICON (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411142398.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing server monitoring systems generate a large amount of data in large-scale environments, resulting in large resource consumption and complex management, making it difficult to effectively monitor server usage, which may lead to important information being ignored and affecting problem diagnosis and resolution.

Method used

By obtaining the server's historical monitoring data, training the prediction model, combining real-time monitoring data to dynamically adjust the monitoring frequency, and alarming the user when the difference exceeds the preset threshold, machine learning algorithms such as random forest and LSTM models are used to construct feature data sets and optimize Model parameters to improve prediction accuracy.

Benefits of technology

Improve resource efficiency, realize timely abnormal detection and response, reduce operation and maintenance costs, improve system operation efficiency and resource utilization, enhance monitoring and response capabilities for abnormal states, and improve system reliability and stability sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939508A_ABST
    Figure CN118939508A_ABST
Patent Text Reader

Abstract

The invention provides an intelligent data monitoring method and system for a server, and the method comprises the steps: obtaining historical monitoring data of the server, and training a prediction model according to the historical monitoring data; acquiring real-time monitoring data of the server, and determining a monitoring frequency according to a difference value between the real-time monitoring data and the data predicted by the prediction model; and monitoring the service condition of the server according to the monitoring frequency, and when the difference value between the data monitored in real time and the data predicted by the prediction model exceeds a preset threshold value, giving an alarm to a user. Through the method and the corresponding system, unnecessary data processing and resource consumption can be effectively reduced, so that the overall efficiency of the system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention proposes a server intelligent data monitoring method and system, belonging to the technical field of data processing and monitoring. Background Art

[0002] Server monitoring is essential to check the health and performance of the server as an unnoticed failure can even cause potential damage to the network and business operations. Server monitoring carefully checks server availability, capacity, resource load, etc. to achieve faster data transfer and prevent data or business loss.

[0003] Monitoring systems often generate a large amount of data, especially in large-scale environments. Processing and analyzing this massive amount of data often requires a lot of resources and is difficult to manage. In addition, data overload may cause important information to be overlooked, making problem diagnosis and resolution more complicated and failing to effectively monitor server usage. Summary of the invention

[0004] The present invention provides a server intelligent data monitoring method and system to solve the above-mentioned problems: The present invention provides a server intelligent data monitoring method, the method comprising: Obtaining historical monitoring data of the server, and training a prediction model based on the historical monitoring data; Obtain real-time monitoring data of the server, and determine the monitoring frequency based on the difference between the real-time monitoring data and the data predicted by the prediction model; The server usage is monitored according to the monitoring frequency. When the difference between the real-time monitored data and the data predicted by the prediction model exceeds the preset threshold, an alarm is sent to the user.

[0005] Furthermore, historical monitoring data of the server is obtained, and a prediction model is trained according to the historical monitoring data, including: Collect historical monitoring data of the server from the server log and monitoring system, and the historical monitoring data of the server includes: CPU usage, memory usage, disk usage, and network traffic; Preprocessing the collected data after collection, wherein the preprocessing includes removing and correcting damaged, incomplete and abnormal data points in the historical monitoring data of the server; After preprocessing, the data is standardized, and the historical monitoring data of the standardized server is grouped by time window, and the statistics of the historical monitoring data of each time window are calculated, and the statistics include the average value, maximum value, minimum value, median and standard deviation, and the time characteristics of the corresponding time window are extracted; The importance of the statistical and temporal features is evaluated by using a machine learning algorithm, Random Forest, to select the features that contribute most to the model's predictive ability; Creating new features based on existing time window statistics, the new features include dynamic features based on sliding windows, growth rate and trend line features, and time difference features; Construct a feature dataset by corresponding the most contributing features, new features and time series, and divide the feature dataset into training set, test set and validation set; The LSTM model is trained using the training set, the performance of the LSTM model is tested on the validation set and the hyperparameters are adjusted, the model effectiveness is evaluated using the test set, and the parameters and structure of the LSTM model are adjusted and optimized according to the test results to obtain the final prediction model.

[0006] Furthermore, real-time monitoring data of the server is obtained, and the monitoring frequency is determined according to the difference between the real-time monitoring data and the data predicted by the prediction model, including: Continuously collect server operation data; Pre-processing the collected operation data, wherein the pre-processing includes data cleaning, formatting and normalization; Input the pre-processed data into the prediction model to predict the CPU usage, memory usage, disk usage and network traffic of the server in the current period; Compare the data output by the prediction model with the actual monitoring data to obtain the quantitative difference between the data output by the prediction model and the actual monitoring data; The monitoring frequency of the next time period is determined according to the quantity difference through the monitoring frequency model. Specifically, the monitoring frequency model is:

[0007] Where Δ is the quantity difference, a is the basic sampling frequency, and b, c, d, e, g, and h are adjustment coefficients.

[0008] Furthermore, the server usage is monitored according to the monitoring frequency. When the difference between the real-time monitored data and the data predicted by the prediction model exceeds a preset threshold, an alarm is given to the user, including: Monitor server usage based on monitoring frequency; Input the data monitored at the monitoring frequency into the prediction model to obtain prediction data; The difference between the predicted data and the real-time monitoring data of the time period corresponding to the predicted data is compared. If the difference exceeds the preset threshold, an alarm message is sent to the user.

[0009] The present invention provides a server intelligent data monitoring system, the system comprising: A training model module is used to obtain historical monitoring data of the server and train a prediction model based on the historical monitoring data; A monitoring frequency determination module is used to obtain real-time monitoring data of the server and determine the monitoring frequency according to the difference between the real-time monitoring data and the data predicted by the prediction model; The alarm module is used to monitor the server usage according to the monitoring frequency, and to alert the user when the difference between the real-time monitored data and the data predicted by the prediction model exceeds a preset threshold.

[0010] Furthermore, the training model module includes: The module for obtaining historical monitoring data is used to collect historical monitoring data of the server from the server log and the monitoring system. The historical monitoring data of the server includes: CPU usage, memory usage, disk usage, and network traffic; A preprocessing module, used for preprocessing the collected data, wherein the preprocessing includes removing and correcting damaged, incomplete and abnormal data points in the historical monitoring data of the server; A statistics calculation module is used to perform standardization on the preprocessed data, group the standardized historical monitoring data of the server by time window, calculate the statistics of the historical monitoring data of each time window, the statistics include the average value, maximum value, minimum value, median and standard deviation, and extract the time characteristics of the corresponding time window; Select the most contributing feature module, which is used to evaluate the importance of the statistics and time features through the machine learning algorithm random forest, and select the feature that contributes most to the model prediction ability; Constructing a new feature module for creating new features based on existing time window statistics, wherein the new features include dynamic features based on sliding windows, growth rate and trend line features, and time difference features; The feature data set division module is used to construct a feature data set corresponding to the most contributing features, new features and time series, and divide the feature data set into a training set, a test set and a validation set; The training final model module is used to train the LSTM model using the training set, test the performance of the LSTM model on the validation set and adjust the hyperparameters, evaluate the model effectiveness using the test set, and adjust and optimize the LSTM model parameters and structure according to the test results to obtain the final prediction model.

[0011] Further, the module for determining the monitoring frequency includes: Data collection module, used to continuously collect server operation data; A pre-processing module, used for pre-processing the collected operation data, wherein the pre-processing includes data cleaning, formatting and normalization; A prediction module, used to input the pre-processed data into the prediction model to predict the CPU usage, memory usage, disk usage and network traffic of the server in the current period; A quantity difference obtaining module is used to compare the data output by the prediction model with the actual monitoring data to obtain the quantity difference between the data output by the prediction model and the actual monitoring data; The monitoring frequency calculation module is used to determine the monitoring frequency of the next time period according to the quantity difference through the monitoring frequency model. Specifically, the monitoring frequency model is:

[0012] Where Δ is the quantity difference, a is the basic sampling frequency, and b, c, d, e, g, and h are adjustment coefficients.

[0013] Furthermore, in a server intelligent data monitoring system, the alarm module comprises: A monitoring module, used to monitor server usage according to a monitoring frequency; A prediction data acquisition module is used to input the data monitored according to the monitoring frequency into the prediction model to obtain prediction data; The alarm information sending module is used to compare the difference between the predicted data and the real-time monitoring data of the time period corresponding to the predicted data. If the difference exceeds a preset threshold, an alarm message is sent to the user.

[0014] The beneficial effects of the present invention are as follows: improving resource efficiency, by dynamically adjusting the monitoring frequency according to the deviation between real-time data and predicted results, unnecessary data processing and resource consumption can be effectively reduced, thereby improving the overall efficiency of the system; timely anomaly detection and response, the solution can promptly discover and report anomalies in server behavior, which is critical for maintaining stable operation of the system and timely responding to potential problems; in the continuous monitoring process, the prediction model can be continuously updated and optimized through newly collected data, making the model more accurate and adaptable to the current system status; reducing operation and maintenance costs, the automated monitoring and early warning system reduces the need for manual intervention, reduces operation and maintenance costs, and allows system administrators to focus on more important tasks; the technical solution uses intelligent data processing and monitoring methods to not only improve the system's operating efficiency and resource utilization, but also strengthen the monitoring and response capabilities to abnormal conditions, reduce overall operation and maintenance costs, and improve the reliability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a server intelligent data monitoring method described in the present invention. DETAILED DESCRIPTION

[0016] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0017] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0019] One embodiment of the present invention provides a server intelligent data monitoring method, the method comprising: Obtaining historical monitoring data of the server, and training a prediction model based on the historical monitoring data; Obtain real-time monitoring data of the server, and determine the monitoring frequency based on the difference between the real-time monitoring data and the data predicted by the prediction model; The server usage is monitored according to the monitoring frequency. When the difference between the real-time monitored data and the data predicted by the prediction model exceeds the preset threshold, an alarm is sent to the user.

[0020] The working principle and effect of the above technical solution are as follows: historical monitoring data acquisition and prediction model training. First, the historical monitoring data of the server is collected. These data include key performance indicators such as the server's CPU usage, memory usage, disk I / O, network traffic, etc. These historical data are used to train a prediction model. Machine learning algorithms (such as linear regression, decision trees, support vector machines, neural networks, etc.) are usually used. The purpose of the model is to learn patterns and trends in historical data in order to predict the status of the server at a certain moment in the future; real-time monitoring data acquisition and dynamic adjustment of monitoring frequency. The system continuously acquires real-time monitoring data from the server and inputs this data into the previously trained prediction model; the model uses these real-time data to predict the next server performance and compares the prediction results with the real-time data. The monitoring frequency is dynamically adjusted based on the difference (deviation) between the two. If the difference is small, it indicates that the server is safe and the monitoring frequency can be reduced. Otherwise, the monitoring frequency is increased to ensure that key changes can be captured. When the difference between the real-time monitoring data and the model prediction data exceeds the preset threshold, the system will determine it as an abnormal state. This threshold is set based on historical monitoring data and may take into account factors such as data volatility and business criticality. Once an anomaly is detected, the system will send an alarm to the user or system administrator to remind them to check possible problems or take appropriate measures. Improve resource efficiency. By dynamically adjusting the monitoring frequency according to the deviation between real-time data and predicted results, unnecessary data processing and resource consumption can be effectively reduced, thereby improving the overall efficiency of the system. Timely anomaly detection and response. The solution can promptly detect and report anomalies in server behavior, which is critical for maintaining stable system operation and timely responding to potential problems. In the continuous monitoring process, the prediction model can be continuously updated and optimized through newly collected data to make the model more accurate and adaptable to the current system status. Reduce operation and maintenance costs. The automated monitoring and early warning system reduces the need for manual intervention, reduces operation and maintenance costs, and allows system administrators to focus on more important tasks. Through intelligent data processing and monitoring methods, this technical solution not only improves the system's operating efficiency and resource utilization, but also strengthens the monitoring and response capabilities to abnormal conditions, reduces overall operation and maintenance costs, and improves system reliability and stability.

[0021] One embodiment of the present invention is a server intelligent data monitoring method, which obtains historical monitoring data of the server and trains a prediction model based on the historical monitoring data, including: Collect historical monitoring data of the server from the server log and monitoring system, and the historical monitoring data of the server includes: CPU usage, memory usage, disk usage, and network traffic; Preprocessing the collected data after collection, wherein the preprocessing includes removing and correcting damaged, incomplete and abnormal data points in the historical monitoring data of the server; After preprocessing, the data is standardized, and the historical monitoring data of the standardized server is grouped by time window, and the statistics of the historical monitoring data of each time window are calculated, and the statistics include the average value, maximum value, minimum value, median and standard deviation, and the time characteristics of the corresponding time window are extracted; The importance of the statistical and temporal features is evaluated by using a machine learning algorithm, Random Forest, to select the features that contribute most to the model's predictive ability; Creating new features based on existing time window statistics, the new features including dynamic features based on sliding windows, growth rate and trend line features, and time difference features; Construct a feature dataset by corresponding the most contributing features, new features and time series, and divide the feature dataset into training set, test set and validation set; The LSTM model is trained using the training set, the performance of the LSTM model is tested on the validation set and the hyperparameters are adjusted, the model effectiveness is evaluated using the test set, and the parameters and structure of the LSTM model are adjusted and optimized according to the test results to obtain the final prediction model.

[0022] The working principle and effect of the above technical solution are as follows: data collection and preprocessing, obtaining historical monitoring data from server logs and monitoring systems, including CPU usage, memory usage, disk usage and network traffic, preprocessing, removing and correcting damaged, incomplete or abnormal data points, and ensuring the accuracy of subsequent analysis; data standardization and feature engineering, standardizing the preprocessed data to ensure the comparability of data in different dimensions; grouping the data by time windows, and calculating the statistics in each window (such as mean, maximum, minimum, median, standard deviation) and extracting time features; using random forests to evaluate the importance of each statistic and time feature, and selecting the features that contribute most to the model's predictive ability; generating new features, such as dynamic features based on sliding windows, growth rate and trend line features, and time difference features; model training and evaluation, integrating the selected features into a feature data set, and dividing it into training set, test set and validation set; using the training set to train the long short-term memory network (LSTM model). The LSTM model can learn the long-term dependencies of time series data. The performance of the LSTM model is tested on the validation set, and the model's hyperparameters are adjusted. The performance of the final model is evaluated using the test set, and the model parameters and structure are adjusted and optimized based on the test results to obtain the final prediction model. Improve prediction accuracy. Through meticulous feature engineering and the use of complex models such as LSTM that are suitable for processing time series, the prediction accuracy of server status is significantly improved. Increase the generalization ability of the model. After precise feature selection and strict test set evaluation, the generalization ability of the model will be enhanced, making its performance on unseen data more stable. Real-time monitoring and dynamic adjustment. The model can perform dynamic predictions based on real-time data, helping system administrators to promptly identify potential problems and respond quickly. Preventive maintenance and reducing downtime. The predictive model can detect possible problems in the system in advance, thereby implementing preventive maintenance and reducing the system's unexpected downtime; optimizing resource allocation. Through accurate server performance prediction, resource allocation and load distribution can be arranged more reasonably, improving server operation efficiency and cost-effectiveness; this technical solution can be widely used in the management of modern IT infrastructure, and through intelligent data analysis and machine learning models, it can significantly improve the quality and efficiency of server operation and maintenance.

[0023] An embodiment of the present invention is a server intelligent data monitoring method, which obtains real-time monitoring data of the server and determines the monitoring frequency according to the difference between the real-time monitoring data and the data predicted by the prediction model, including: Continuously collect server operation data; Pre-processing the collected operation data, wherein the pre-processing includes data cleaning, formatting and normalization; Input the pre-processed data into the prediction model to predict the CPU usage, memory usage, disk usage and network traffic of the server in the current period; Compare the data output by the prediction model with the actual monitoring data to obtain the quantitative difference between the data output by the prediction model and the actual monitoring data; The monitoring frequency of the next time period is determined according to the quantity difference through the monitoring frequency model. Specifically, the monitoring frequency model is:

[0024] Among them, Δ is the difference in quantity, a is the basic sampling frequency, that is, the minimum hold frequency when the change value is very small, b, c, d, e, g and h are adjustment coefficients, Δ is the change in the monitored value, b, c, d, e, g, h are adjustment coefficients, each of which has its own unique role to control the details and sensitivity of the frequency adjustment, exp(⋅) is an exponential function to provide a fast response in a small change value range, and tanh(⋅) is a hyperbolic tangent function, which is used to smoothly adjust the frequency response when the change value is large. a: Select the minimum frequency that is sufficient to capture important information but does not cause redundancy; b: Control the maximum change in frequency within a very small change value range; c: Control the increase in frequency when the change value is large; d, e: Affect the speed and sensitivity of the exponential decay; g: Affect the slope of the hyperbolic tangent part, that is, the sensitivity of the change value to the frequency; h: Adjust at what level of change value to start significantly increasing the sampling frequency.

[0025] The value range of b is [2,6], the value range of c is [1,5], the value range of d is [0.5,2], the value range of e is [0.1,1], the value range of g is [0.05,2], and the value range of h is [0.5,2].

[0026] The main purpose of this formula design is to dynamically adjust the sampling frequency to adapt to changes in server operation data. In this way, it can respond more intelligently to the actual needs of the system, optimize resource usage, and improve system efficiency and stability. A minimum monitoring frequency is set to keep the system from being able to continuously monitor the operating status when the change is small, ensuring that key data is not missed. Exponential decay part (b * exp(-1 / (d + e * |Δ|))): When the difference (Δ) is small, that is, the predicted value is very close to the actual value, the term in exp increases, resulting in a decrease in the contribution of this part, which is reflected in a decrease in the total sampling frequency. This exponential decay ensures that small changes do not lead to large changes in the sampling frequency, which helps save resources and reduce unnecessary sampling. The coefficients d and e allow fine-tuning of the rate and sensitivity of the exponential decay to adapt to different rates of change. The hyperbolic tangent part (c * tanh(g * |Δ| - h)): When the difference becomes larger, the output of the hyperbolic tangent function approaches its maximum value, resulting in an increase in the contribution of this part, thereby increasing the overall sampling frequency. This mechanism ensures that when a large deviation occurs, the system can respond quickly by increasing the sampling frequency to better monitor and resolve emergencies. The coefficients g and h The slope and horizontal displacement of the control function affect the sensitivity and threshold of the frequency increase; the balance between response speed and sensitivity, by combining exponential decay and hyperbolic tangent function, the formula can respond quickly to large changes sensitively, while avoiding overreaction to small changes, keeping the system running smoothly; adaptive dynamic adjustment, the formula can automatically adjust the sampling frequency according to the changes in real-time monitoring data to achieve true dynamic monitoring. This adaptability is very important, especially when dealing with complex and constantly changing systems, and can effectively manage resources and optimize performance; dynamic adjustment of the sampling frequency helps to avoid unnecessary data collection and processing, thereby reducing storage and computing costs, improving the overall resource utilization efficiency of the system, and improving the quality and accuracy of data processing. By ensuring that the sampling frequency is increased when needed, the system status can be monitored and analyzed more accurately, and the quality and availability of data can be improved; this design not only improves the responsiveness and stability of the system through intelligent adjustment of the monitoring strategy, but also optimizes resource utilization and improves the overall efficiency and effectiveness of monitoring activities.

[0027] An embodiment of the present invention is a server intelligent data monitoring method, which monitors server usage according to a monitoring frequency, and alerts a user when the difference between the real-time monitored data and the data predicted by the prediction model exceeds a preset threshold, including: Monitor server usage based on monitoring frequency; Input the data monitored at the monitoring frequency into the prediction model to obtain prediction data; The difference between the predicted data and the real-time monitoring data of the time period corresponding to the predicted data is compared. If the difference exceeds the preset threshold, an alarm message is sent to the user.

[0028] One embodiment of the present invention provides a server intelligent data monitoring system, the system comprising: A training model module is used to obtain historical monitoring data of the server and train a prediction model based on the historical monitoring data; A monitoring frequency determination module is used to obtain real-time monitoring data of the server and determine the monitoring frequency according to the difference between the real-time monitoring data and the data predicted by the prediction model; The alarm module is used to monitor the server usage according to the monitoring frequency, and to alert the user when the difference between the real-time monitored data and the data predicted by the prediction model exceeds a preset threshold.

[0029] In one embodiment of the present invention, a server intelligent data monitoring system, the training model module includes: The module for obtaining historical monitoring data is used to collect historical monitoring data of the server from the server log and the monitoring system. The historical monitoring data of the server includes: CPU usage, memory usage, disk usage, and network traffic; A preprocessing module, used for preprocessing the collected data, wherein the preprocessing includes removing and correcting damaged, incomplete and abnormal data points in the historical monitoring data of the server; A statistics calculation module is used to perform standardization on the preprocessed data, group the standardized historical monitoring data of the server by time window, calculate the statistics of the historical monitoring data of each time window, the statistics include the average value, maximum value, minimum value, median and standard deviation, and extract the time characteristics of the corresponding time window; Select the most contributing feature module, which is used to evaluate the importance of the statistics and time features through the machine learning algorithm random forest, and select the feature that contributes most to the model prediction ability; Constructing a new feature module for creating new features based on existing time window statistics, wherein the new features include dynamic features based on sliding windows, growth rate and trend line features, and time difference features; The feature data set division module is used to construct a feature data set corresponding to the most contributing features, new features and time series, and divide the feature data set into a training set, a test set and a validation set; The training final model module is used to train the LSTM model using the training set, test the performance of the LSTM model on the validation set and adjust the hyperparameters, evaluate the model effectiveness using the test set, and adjust and optimize the LSTM model parameters and structure according to the test results to obtain the final prediction model.

[0030] In one embodiment of the present invention, a server intelligent data monitoring system, the module for determining monitoring frequency includes: Data collection module, used to continuously collect server operation data; A pre-processing module, used for pre-processing the collected operation data, wherein the pre-processing includes data cleaning, formatting and normalization; A prediction module, used to input the pre-processed data into the prediction model to predict the CPU usage, memory usage, disk usage and network traffic of the server in the current period; A quantity difference obtaining module is used to compare the data output by the prediction model with the actual monitoring data to obtain the quantity difference between the data output by the prediction model and the actual monitoring data; The monitoring frequency calculation module is used to determine the monitoring frequency of the next time period according to the quantity difference through the monitoring frequency model. Specifically, the monitoring frequency model is:

[0031] Where Δ is the quantity difference, a is the basic sampling frequency, b, c, d, e, g and h are adjustment coefficients In one embodiment of the present invention, a server intelligent data monitoring system, the alarm module comprises: A monitoring module, used to monitor server usage according to a monitoring frequency; A prediction data acquisition module is used to input the data monitored according to the monitoring frequency into the prediction model to obtain prediction data; The alarm information sending module is used to compare the difference between the predicted data and the real-time monitoring data of the time period corresponding to the predicted data. If the difference exceeds a preset threshold, an alarm message is sent to the user.

[0032] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A server intelligent data monitoring method, characterized in that: The method comprises: Obtaining historical monitoring data of the server, and training a prediction model based on the historical monitoring data; Obtain real-time monitoring data of the server, and determine the monitoring frequency based on the difference between the real-time monitoring data and the data predicted by the prediction model; The server usage is monitored according to the monitoring frequency. When the difference between the real-time monitored data and the data predicted by the prediction model exceeds the preset threshold, an alarm is sent to the user.

2. A server intelligent data monitoring method according to claim 1, characterized in that: Obtaining historical monitoring data of the server and training a prediction model based on the historical monitoring data includes: Collect historical monitoring data of the server from the server log and monitoring system, and the historical monitoring data of the server includes: CPU usage, memory usage, disk usage, and network traffic; Preprocessing the collected data after collection, wherein the preprocessing includes removing and correcting damaged, incomplete and abnormal data points in the historical monitoring data of the server; After preprocessing, the data is standardized, and the historical monitoring data of the standardized server is grouped by time window, and the statistics of the historical monitoring data of each time window are calculated, and the statistics include the average value, maximum value, minimum value, median and standard deviation, and the time characteristics of the corresponding time window are extracted; The importance of the statistical and temporal features is evaluated by using a machine learning algorithm, Random Forest, to select the features that contribute most to the model's predictive ability; Creating new features based on existing time window statistics, the new features including dynamic features based on sliding windows, growth rate and trend line features, and time difference features; Construct a feature dataset by corresponding the most contributing features, new features and time series, and divide the feature dataset into training set, test set and validation set; The LSTM model is trained using the training set, the performance of the LSTM model is tested on the validation set and the hyperparameters are adjusted, the model effectiveness is evaluated using the test set, and the parameters and structure of the LSTM model are adjusted and optimized according to the test results to obtain the final prediction model.

3. A server intelligent data monitoring method according to claim 1, characterized in that: Obtain the real-time monitoring data of the server and determine the monitoring frequency based on the difference between the real-time monitoring data and the data predicted by the prediction model, including: Continuously collect server operation data; Pre-processing the collected operation data, wherein the pre-processing includes data cleaning, formatting and normalization; Input the pre-processed data into the prediction model to predict the CPU usage, memory usage, disk usage and network traffic of the server in the current period; Compare the data output by the prediction model with the actual monitoring data to obtain the quantitative difference between the data output by the prediction model and the actual monitoring data; The monitoring frequency of the next time period is determined according to the quantity difference through the monitoring frequency model. Specifically, the monitoring frequency model is: Where Δ is the quantity difference, a is the basic sampling frequency, and b, c, d, e, g, and h are adjustment coefficients.

4. A server intelligent data monitoring method according to claim 1, characterized in that: Monitor server usage based on the monitoring frequency. When the difference between the real-time monitoring data and the data predicted by the prediction model exceeds the preset threshold, an alarm is issued to the user, including: Monitor server usage based on monitoring frequency; Input the data monitored at the monitoring frequency into the prediction model to obtain prediction data; The difference between the predicted data and the real-time monitoring data of the time period corresponding to the predicted data is compared. If the difference exceeds the preset threshold, an alarm message is sent to the user.

5. A server intelligent data monitoring system, characterized in that: The system comprises: A training model module is used to obtain historical monitoring data of the server and train a prediction model based on the historical monitoring data; A monitoring frequency determination module is used to obtain real-time monitoring data of the server and determine the monitoring frequency according to the difference between the real-time monitoring data and the data predicted by the prediction model; The alarm module is used to monitor the server usage according to the monitoring frequency, and to alert the user when the difference between the real-time monitored data and the data predicted by the prediction model exceeds a preset threshold.

6. A server intelligent data monitoring system according to claim 5, characterized in that: The training model module includes: The module for obtaining historical monitoring data is used to collect historical monitoring data of the server from the server log and the monitoring system. The historical monitoring data of the server includes: CPU usage, memory usage, disk usage, and network traffic; A preprocessing module, used for preprocessing the collected data, wherein the preprocessing includes removing and correcting damaged, incomplete and abnormal data points in the historical monitoring data of the server; A statistics calculation module is used to perform standardization on the preprocessed data, group the standardized historical monitoring data of the server by time window, calculate the statistics of the historical monitoring data of each time window, the statistics include the average value, maximum value, minimum value, median and standard deviation, and extract the time characteristics of the corresponding time window; Select the most contributing feature module, which is used to evaluate the importance of the statistics and time features through the machine learning algorithm random forest, and select the feature that contributes most to the model prediction ability; Constructing a new feature module for creating new features based on existing time window statistics, wherein the new features include dynamic features based on sliding windows, growth rate and trend line features, and time difference features; The feature data set division module is used to construct a feature data set corresponding to the most contributing features, new features and time series, and divide the feature data set into a training set, a test set and a validation set; The training final model module is used to train the LSTM model using the training set, test the performance of the LSTM model on the validation set and adjust the hyperparameters, evaluate the model effectiveness using the test set, and adjust and optimize the LSTM model parameters and structure according to the test results to obtain the final prediction model.

7. A server intelligent data monitoring system according to claim 5, characterized in that: The module for determining the monitoring frequency includes: Data collection module, used to continuously collect server operation data; A pre-processing module, used for pre-processing the collected operation data, wherein the pre-processing includes data cleaning, formatting and normalization; A prediction module, used to input the pre-processed data into the prediction model to predict the CPU usage, memory usage, disk usage and network traffic of the server in the current period; A quantity difference obtaining module is used to compare the data output by the prediction model with the actual monitoring data to obtain the quantity difference between the data output by the prediction model and the actual monitoring data; The monitoring frequency calculation module is used to determine the monitoring frequency of the next time period according to the quantity difference through the monitoring frequency model. Specifically, the monitoring frequency model is: Where Δ is the quantity difference, a is the basic sampling frequency, and b, c, d, e, g, and h are adjustment coefficients.

8. A server intelligent data monitoring system according to claim 5, characterized in that: The alarm module comprises: A monitoring module, used to monitor server usage according to a monitoring frequency; A prediction data acquisition module is used to input the data monitored according to the monitoring frequency into the prediction model to obtain prediction data; The alarm information sending module is used to compare the difference between the predicted data and the real-time monitoring data of the time period corresponding to the predicted data. If the difference exceeds a preset threshold, an alarm message is sent to the user.

Citation Information

Patent Citations

  • Equipment data acquisition and monitoring method and system based on SNMP (Simple Network Management Protocol)

    CN117596119A

  • Small electronic equipment data wireless communication method based on Internet of Things

    CN117641398A

  • Training method and device for failure prediction neural network model

    WO2022048168A1