Server cluster abnormity early warning system and method based on fusion technology

By collecting performance and environmental data in the server cluster, using feature filtering REF algorithm and multiple early warning methods for abnormal detection, the problem of insufficient early warning content, methods and results in the existing technology is solved, and detailed and accurate abnormal warning is achieved. Reports improve managers' coping skills.

CN119003227APending Publication Date: 2024-11-22JIANGSU HUAKUN ZHENYU INTELLIGENT TECH CO LTD +1

Patent Information

Application Number
CN202411489080.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the server cluster abnormal warning, the existing technology has problems such as insufficient early warning content, insufficient early warning methods and insufficient early warning results, which leads to the inability to deal with abnormal abnormalities in a timely, accurate and effective manner.

Method used

Using the server cluster exception warning method based on converged technology, by collecting the initial feature set, including performance indicators and environmental data, the feature filtering REF algorithm model selects a subset of features related to the abnormal state, and based on multiple early warning methods (such as rules, Machine learning, deep learning) performs abnormal detection and outputs detailed warning reports.

Benefits of technology

Improve the accuracy and coverage of abnormal detection, provide detailed warning reports, including exception type, level, location, causes and recommended measures, enhance managers' decision-making capabilities and adapt to server cluster environments of different types and sizes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003227A_ABST
    Figure CN119003227A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of computer fault diagnosis, and particularly relates to a server cluster abnormity early warning system and method based on a fusion technology, an initial feature set of a server cluster is collected, and the initial feature set comprises performance indexes and environment data of the server cluster; selecting a feature subset related to the abnormal state of the server cluster from the initial feature set through a feature screening REF algorithm model; performing anomaly detection on the feature subset based on a fusion method; and based on an anomaly detection result, judging whether an anomaly exists or not, if so, outputting an anomaly early warning report, and if not, continuously collecting server cluster features. The most relevant feature subset is automatically selected through the REF algorithm, the influence of irrelevant or redundant information is reduced, and the efficiency and performance of the model are improved. By acquiring various performance indexes and environmental factors of the server cluster, the running state of the server cluster is comprehensively monitored, and potential and already occurring abnormal and risk events are found in time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer fault diagnosis, and in particular relates to a server cluster abnormality early warning system and method based on fusion technology. Background Art

[0002] A server cluster is a system composed of multiple servers that provides high-performance, high-availability, and high-scalability services. However, server clusters are also subject to various risks of failure, such as hardware failure, software failure, network failure, configuration error, attack, etc. These failures may lead to serious consequences such as service interruption, performance degradation, data loss or leakage. Therefore, abnormal early warning of server clusters is an important means to ensure the normal operation and security of server clusters.

[0003] At present, there are some technologies used to provide abnormal warnings for server clusters, such as rule-based warning technology, support vector machine-based warning technology, long short-term memory network-based warning technology, and log and event-based warning technology. However, each technology has its limitations and shortcomings, which are mainly manifested in the following aspects: Insufficient warning content. Existing technologies usually only consider some characteristics of the server cluster, such as performance indicators or log events, while ignoring other characteristics that may reflect abnormal conditions, such as environmental factors, user behavior, network topology, etc. This may cause some hidden or complex anomalies to go undetected, or result in false positives or missed positives.

[0004] Inadequate early warning methods. Existing technologies usually only use one or a few early warning methods, and do not fully utilize the advantages of multiple early warning methods, such as rules, machine learning, deep learning, natural language processing, graph analysis, etc. This may cause some early warning methods to be unable to adapt to the dynamic changes of server clusters, or unable to handle different types or levels of anomalies, or unable to provide sufficient early warning information and suggestions.

[0005] Insufficient early warning results. Existing technologies usually only output simple early warning results, such as the presence or absence of anomalies, the category or level of anomalies, etc., without integrating the results of multiple early warning methods, or providing detailed early warning reports such as the location, cause, impact range, and recommended measures of the anomaly. This may result in the inability of server cluster managers to handle anomalies in a timely, accurate, and effective manner, or to prevent the occurrence or spread of anomalies.

[0006] Therefore, how to overcome the problems of insufficient warning content, insufficient warning methods and insufficient warning results in the existing technology is a technical problem that urgently needs to be solved. Summary of the invention

[0007] The purpose of the present invention is to overcome the technical problems in the prior art of insufficient warning content, insufficient warning method and insufficient warning result.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, a server cluster abnormality warning method based on fusion technology is provided, comprising the following steps: S1: Collecting an initial feature set of a server cluster, wherein the initial feature set includes various performance indicators and environmental data of the server cluster; S2: selecting a feature subset related to the abnormal state of the server cluster from the initial feature set through a feature screening REF algorithm model; S3: performing anomaly detection on the feature subset based on a fusion method; S4: Based on the anomaly detection result, determine whether there is an anomaly. If so, execute step S5. If not, continue to collect server cluster features and execute step S1 repeatedly. S5: Output abnormal warning report, including abnormal type, level, location, cause, impact range and recommended handling method.

[0009] Preferably, the performance indicators of the server cluster in step S1 include response time, throughput, server resources, network bandwidth, software configuration and hardware configuration, CPU utilization, memory utilization, disk I / O, network traffic, network latency, radiator performance parameters, fluctuation of service response time, application performance, and network topology; the environmental data include server temperature, data center temperature, and ambient temperature and humidity.

[0010] Preferably, the performance indicators of the server cluster are monitored and collected in real time by monitoring software installed on the server cluster, and the server operation status and user access records are obtained by analyzing the log files of the server cluster; By deploying sensors in the server cluster, the server temperature and ambient temperature and humidity are monitored.

[0011] Preferably, step S2 includes the following specific process: S21: performing data preprocessing on the collected initial feature set of the server cluster; S22: dividing the preprocessed initial feature set of the server cluster into a training set, a test set, and a validation set, which serve as training input, test input, and validation input of the feature screening model; S23: Construct a random forest model as a feature screening model to perform feature subset screening.

[0012] Preferably, step S21 includes the following specific process: S211: De-noising: removing random noise in the initial feature set; S212: Identify abnormal data in the initial feature set, including missing values ​​and outliers; S213: Missing value and outlier processing: Processing missing value and outlier data in the initial feature set; S214: Normalization processing: converting the data in the initial feature set after noise removal into a form with a similar scale; The initial feature set of the server cluster is expressed as: ; Among them, k represents the different labels or sources of the collected data, and n represents the dimension of the data, that is, the feature values ​​at different times. represents the i-th data point of the j-th type; The formula for removing noise in step S211 is: ; in, x ( i ) indicates the i data points, t represents the position or index of the data point to be smoothed or filtered, and is an integer representing the position of a time point or data point in the time series; The missing value processing formula in step S213 is: ; in, For missing values For missing values The previous data, For missing values The latter data, is the sequence number of the data; The abnormal value processing formula in step S213 is: ; in, is the mean of x(n), is the standard deviation of x(n), Z i ( x ) is greater than the custom threshold, indicating that i Data points x ( i ) deviates too far from the mean and is marked as an outlier.

[0013] Preferably, the feature screening in step S23 includes the following process: S23: constructing a random forest model, and training the random forest model using a training set; S24: Use the trained random forest model to predict the test set and calculate the importance of each feature; S25: Sort the features according to the importance of each feature; In step S24, the calculation formula of the importance of the feature is: ; in, Representation characteristics f The importance of, T represents the number of decision trees in the random forest, It(f) represents the feature in the tth decision tree f The importance of It(f) can be expressed by the following formula: ; Among them, among them, N t ( f ) indicates that the t-th decision tree contains the feature f The collection of nodes, N n Indicates n The number of samples of nodes, N t represents the total number of samples in the t-th decision tree, G n Indicates n The Gini index of the nodes; The formula for the Gini index is as follows: ; Among them, K represents the number of categories, Represents the proportion of samples belonging to the kth category in the nth node; The process of sorting the features in step S25 is: select the first m most important features as the feature subset, where m is a hyperparameter that can be adjusted according to actual conditions. The goal of selecting the feature subset is to minimize the loss function of the abnormal warning, which can be expressed by the following formula: ; Among them, P represents the accuracy of the warning, R represents the recall rate of the warning, and α and β are two weight parameters.

[0014] Preferably, the anomaly detection of the fusion method in step S3 is rule-based anomaly detection, machine learning-based anomaly detection, long short-term memory network-based anomaly detection, or log and event-based anomaly detection: Rule-based anomaly detection includes setting a threshold. When the parameters of an important feature exceed the threshold, it is judged as an anomaly and triggers an anomaly warning. Machine learning-based anomaly detection includes support vector machine algorithms; Anomaly detection based on long short-term memory networks captures the long-term dependencies in feature sequence data and learns the time series patterns in the data. In the LSTM layer, multiple LSTM units are set to model the time series information and reshape the obtained features to achieve abnormal target output.

[0015] Secondly, it provides a server cluster abnormality warning system based on fusion technology, including data acquisition module, data preprocessing module, feature screening module, fusion detection module and warning module; The data collection module is used to collect various performance indicators and environmental data of the server cluster and construct an initial feature set; The data preprocessing module is used to preprocess the initial feature set, including cleaning, word segmentation, denoising and normalization; The feature screening module is used to select a feature subset related to the abnormal state of the server cluster from the initial feature set through a feature screening REF algorithm model; The fusion detection module is used to perform anomaly detection on the feature subset based on a fusion method; The warning module is used to integrate the results of the detection module and output an abnormal warning report, including the type, level, location, cause, impact range, and recommended processing method of the abnormality.

[0016] Preferably, the feature screening module adopts a feature screening REF algorithm, which evaluates and sorts the initial feature set based on three indicators: correlation, redundancy and abnormality, and selects the optimal feature subset.

[0017] The beneficial effects of the present invention include: The scalable and highly available containerized server architecture provided by the present invention collects an initial feature set of a server cluster, wherein the initial feature set includes various performance indicators and environmental data of the server cluster; a feature subset related to the abnormal state of the server cluster is selected from the initial feature set through a feature screening REF algorithm model; anomaly detection is performed on the feature subset based on a fusion method; based on the anomaly detection result, it is determined whether there is an anomaly, and if so, an anomaly warning report is output; if not, server cluster feature collection continues.

[0018] 1. The initial feature set includes various performance indicators and environmental data of the server cluster, which is conducive to the subsequent anomaly detection to effectively identify and warn various types of server cluster anomalies.

[0019] 2. A variety of preprocessing and feature screening methods, as well as different machine learning algorithms and deep learning models are used to improve the robustness and generalization ability of the system, and can adapt to server cluster environments of different types and sizes.

[0020] 3. The most relevant feature subset is automatically selected through the REF algorithm, reducing the impact of irrelevant or redundant information and improving the efficiency and performance of the model. Appropriate machine learning or deep learning algorithms can be selected according to specific circumstances, and a variety of log and event analysis technologies are supported, making the system highly flexible and customizable.

[0021] 4. The rule-based early warning technology is simple and intuitive, easy to understand and explain, and can help managers quickly identify abnormal situations and take appropriate measures. At the same time, the system also provides an early warning report module that can clearly display the type, level, location, cause, impact range and recommended measures of the abnormality, helping managers better understand and respond to abnormal situations.

[0022] 5. By acquiring various performance indicators and environmental factors of the server cluster through a variety of data collection methods, it is possible to comprehensively monitor the operating status of the server cluster and promptly discover potential and existing abnormalities and risk events.

[0023] 6. Screen and extract the features of the server cluster from multiple dimensions and angles to effectively reflect the abnormal status of the server cluster, integrate multiple early warning methods to identify and classify the anomalies of the server cluster, and improve the accuracy and coverage of abnormal early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the server cluster abnormality early warning method based on fusion technology of the present invention.

[0025] Figure 2 It is a schematic diagram of the specific process of feature screening of the present invention.

[0026] Figure 3 It is a schematic diagram of the server cluster abnormal warning system based on fusion technology of the present invention. DETAILED DESCRIPTION

[0027] The following is combined with Figure 1-Figure 3 The present invention is further described in detail: Example 1 See also Figure 1 ,The server cluster abnormality warning method based on fusion technology includes the following steps: S1: Collecting an initial feature set of a server cluster, wherein the initial feature set includes various performance indicators and environmental data of the server cluster; S2: selecting a feature subset related to the abnormal state of the server cluster from the initial feature set through a feature screening REF algorithm model; S3: performing anomaly detection on the feature subset based on a fusion method; S4: Based on the anomaly detection result, determine whether there is an anomaly. If so, execute step S5. If not, continue to collect server cluster features and execute step S1 repeatedly. S5: Output abnormal warning report, including abnormal type, level, location, cause, impact range and recommended handling method.

[0028] Collect the initial feature set of the server cluster, which includes the performance indicators and environmental data of the server cluster, and screen and extract the features of the server cluster from multiple dimensions and angles to effectively reflect the abnormal state of the server cluster. Use the feature screening REF algorithm to select the feature subset that best reflects the abnormal state of the server cluster from the initial feature set. This feature subset can minimize the loss function of the abnormal warning, that is, the weighted accuracy and recall rate of the warning.

[0029] By integrating multiple early warning methods, the anomalies of the server cluster can be identified and classified to improve the accuracy and coverage of abnormal early warning. Early warning technology based on rules, support vector machine, long short-term memory network, log and event-based early warning technology, and other possible early warning technologies are also used. According to different abnormal types or levels, select appropriate early warning methods, or combine multiple early warning methods to improve the effectiveness and efficiency of early warning.

[0030] By outputting detailed abnormal warning reports, useful information and suggestions are provided to the managers of the server cluster. Not only can basic information such as the existence of abnormalities, the category or level of abnormalities be output, but also detailed information such as the location, cause, scope of impact, and recommended measures of abnormalities can be output, so that the managers of the server cluster can take corresponding countermeasures in time to ensure the normal operation and security of the server cluster. According to different abnormality types or levels, the format and content of the warning report are adjusted to improve the readability and usability of the warning report.

[0031] In the above scheme, the performance indicators of the server cluster in step S1 include response time, throughput, server resources, network bandwidth, software configuration and hardware configuration, CPU utilization, memory utilization, disk I / O, network traffic, network latency, radiator performance parameters, fluctuation of service response time, application performance, and network topology; the environmental data includes server temperature, data center temperature, and ambient temperature and humidity.

[0032] The performance indicators of the server cluster are monitored and collected in real time through the monitoring software installed on the server cluster. The server operation status and user access records are obtained by analyzing the log files of the server cluster. Sensors are deployed in the server cluster to monitor the temperature of the server and the ambient temperature and humidity.

[0033] Example 2 Based on Example 1, step S2 includes the following specific process: S21: Perform data preprocessing on the collected initial feature set of the server cluster to make the data meet the input requirements of the random forest and prepare the data for further analysis. Because the original data may contain noise, missing values, outliers and other problems, it will affect the accuracy of subsequent server cluster warnings.

[0034] S22: The initial feature set of the preprocessed server cluster is divided into a training set, a test set, and a validation set as the training input, test input, and validation input of the feature screening model. The data set is divided according to the principle of 10-fold cross-validation, where the training set is used to train the learnable parameters in the system and to build a random forest model. The validation set is used to verify the robustness and effectiveness of the system. The test set is used to test the overall objective performance of the algorithm.

[0035] S23: Construct a random forest model as a feature screening model to perform feature subset screening.

[0036] Wherein, step S21 includes the following specific processes: S211: Denoising: remove random noise in the initial feature set to improve data quality; S212: Identify abnormal data in the initial feature set, including missing values ​​and outliers; S213: Missing value and outlier processing: Process missing values ​​and outlier data in the initial feature set to ensure data integrity; S214: Normalization processing: converting the data in the initial feature set after noise removal into a form with similar scale, eliminating the negative impact of sample values ​​of different magnitudes, so that different features can be compared; The initial feature set of the server cluster is expressed as: ; Among them, k is the different labels or sources of the collected data, and n represents the dimension of the data, that is, the feature values ​​at different times. x j ( i ) represents the i-th data point of the j-th type; The formula for removing noise in step S211 is: ; in, x ( i ) represents the i-th data point, t represents the position or index of the data point to be smoothed or filtered, and is an integer representing the position of a time point or data point in the time series; The missing value processing formula in step S213 is: ; in, x ( i ) is a missing value For missing values x ( i )Previous data, For missing values The latter data, i is the sequence number of the data; The abnormal value processing formula in step S213 is: ; in, is the mean of x(n), is the standard deviation of x(n), When it is greater than the custom threshold, it means i Data points If the value deviates too far from the mean, it is marked as an outlier.

[0037] See also Figure 2 , the feature screening in step S23 includes the following process: S23: constructing a random forest model, and training the random forest model using a training set; S24: Use the trained random forest model to predict the test set and calculate the importance of each feature; S25: Sort the features according to the importance of each feature; In step S24, the calculation formula of the importance of the feature is: ; in, Representation characteristics f The importance of, T represents the number of decision trees in the random forest, It(f) represents the feature in the tth decision tree f The importance of.

[0038] It(f) can be expressed by the following formula: ; in, N t ( f) indicates that the t-th decision tree contains the feature f The collection of nodes, N n Indicates n The number of samples of nodes, N t represents the total number of samples in the t-th decision tree, G n Indicates n The Gini index of the nodes.

[0039] The formula for the Gini index is as follows: ; Among them, K represents the number of categories, Indicates n The proportion of samples belonging to the kth category in the nodes.

[0040] Random forest is an integrated model composed of multiple decision trees. Each decision tree is trained on a randomly selected feature subset, which can increase the diversity and robustness of features.

[0041] Example 3 Based on Example 1 or Example 2, the process of sorting the features in step S25 is: selecting the first m most important features as the feature subset, where m is a hyperparameter and can be adjusted according to actual conditions. The goal of selecting the feature subset is to minimize the loss function of the abnormal warning, and the loss function can be expressed by the following formula: ; Among them, P represents the accuracy of the warning, R represents the recall rate of the warning, and α and β are two weight parameters.

[0042] The anomaly detection of the fusion method in step S3 is rule-based anomaly detection, machine learning-based anomaly detection, long short-term memory network-based anomaly detection, or log and event-based anomaly detection: Rule-based anomaly detection includes setting thresholds. When the parameters of an important feature exceed the threshold, it is judged as an anomaly and triggers an anomaly warning.

[0043] Machine learning-based anomaly detection includes support vector machine algorithms.

[0044] By finding a hyperplane, we can effectively separate data of different categories and maximize the interval between categories. In the server cluster abnormality warning task, the support vector machine identifies potential risk events by constructing a decision boundary. The support vector machine is a binary classification algorithm. When the current server cluster abnormality warning is a multi-classification task, multiple binary classifications are performed.

[0045] The goal of a support vector machine is to maximize the margin and find an optimal set of weights and biases such that the following inequality holds: ; Where w is the normal vector of the hyperplane, b is the intercept of the hyperplane, is the input data point, and is the class label of the data point (1 or 0). In order to maximize the margin, the optimization problem of the support vector machine can be expressed as the following convex quadratic programming problem: ; Where st means restricted by. By solving the above optimization problem, an optimal decision boundary is constructed to separate risk and non-risk data. If the result is greater than 0, the data point belongs to the non-risk category, and if the result is less than 0, the data point belongs to the risk category. The goal of training a support vector machine is to minimize the following loss function: ; in, a i is the Lagrange coefficient, n Is the input data x i Length.

[0046] After training is completed, the support vector machine is used to classify new feature points and determine the category of the new data point by calculating the sign of (w⋅x+b). The support vector machine method is an effective method in the server cluster anomaly warning task, which establishes clear decision boundaries between different categories and helps to identify potential risk events in a timely manner.

[0047] Anomaly detection based on long short-term memory network captures the long-term dependencies in feature sequence data and learns the time series pattern in the data. In the LSTM layer, multiple LSTM units are set to model the time series information, reshape the obtained features, and achieve abnormal target output. Several fully connected layers are added after the LSTM layer, pointing to an output as the final prediction result, that is, the predicted server cluster abnormal warning category.

[0048] LSTM is used to select the data to be retained and forgotten, and record the state of the hidden layer. The LSTM unit gates include input gate, forget gate, and output gate. The input gate updates the output of the input gate by using the sigmoid activation function, and the expression is as follows: ; in, i t represents the output of the output gate, and Represent the weight and bias of the input gate respectively, represents the sigmoid activation function, Indicates the last moment The output value of the neuron.

[0049] The forget gate determines the invalid information that is forgotten by the forget unit. This gate uses the tanh function to process the unit state to determine the information to be discarded. It is the current moment t The input of the forget gate is It can be described as: ; ; The value range is [0,1], where 0 means forgetting and 1 means keeping. and are the weight and bias of the forget gate. Indicates the state of the cell. and They represent the weight and bias of the cell respectively, and tanh is the tanh activation function.

[0050] The output gate uses the sigmoid function to update information and control the output, and the tanh function to process the state of the neuron.

[0051] ; ; in, represents the output of the output gate, and denote the weight and bias of the input gate respectively. Represents the output value of the neuron at the current moment.

[0052] See also Figure 3The server cluster abnormal warning system based on fusion technology includes a data acquisition module, a data preprocessing module, a feature screening module, a fusion detection module, and a warning module. The data acquisition module is used to collect various performance indicators and environmental data of the server cluster and construct an initial feature set. The data preprocessing module is used to preprocess the initial feature set, including cleaning, word segmentation, denoising and normalization. The feature screening module is used to select a feature subset related to the abnormal state of the server cluster from the initial feature set through the feature screening REF algorithm model. The fusion detection module is used to perform abnormality detection on the feature subset based on the fusion method. The warning module is used to integrate the results of the fusion detection module and output an abnormal warning report, including the type, level, location, cause, impact range, and recommended processing method of the abnormality. The feature screening module adopts the feature screening REF algorithm, which evaluates and sorts the initial feature set based on the three indicators of correlation, redundancy and abnormality, and selects the optimal feature subset.

[0053] In summary, the scalable and highly available containerized server architecture provided by the present invention collects the initial feature set of the server cluster, and the initial feature set includes various performance indicators and environmental data of the server cluster; through the feature screening REF algorithm model, a feature subset related to the abnormal state of the server cluster is selected from the initial feature set; the feature subset is detected for abnormality based on the fusion method; based on the abnormality detection result, it is determined whether there is an abnormality, and if so, an abnormality warning report is output, if not, the server cluster feature collection continues. The most relevant feature subset is automatically selected by the REF algorithm, reducing the impact of irrelevant or redundant information, thereby improving the efficiency and performance of the model. By obtaining various performance indicators and environmental factors of the server cluster, the operating status of the server cluster is comprehensively monitored, and potential and existing abnormalities and risk events are discovered in a timely manner.

Claims

1. A server cluster abnormality warning method based on fusion technology, characterized in that: The following steps are involved: S1: Collecting an initial feature set of a server cluster, wherein the initial feature set includes various performance indicators and environmental data of the server cluster; S2: Selecting a feature subset related to the abnormal state of the server cluster from the initial feature set through a feature screening REF algorithm model; S3: performing anomaly detection on the feature subset based on a fusion method; S4: Based on the anomaly detection result, determine whether there is an anomaly. If so, execute step S5. If not, continue to collect server cluster features and execute step S1 repeatedly. S5: Output abnormal warning report, including abnormal type, level, location, cause, impact range and recommended handling method.

2. The server cluster abnormality early warning method based on fusion technology according to claim 1 is characterized in that: The performance indicators of the server cluster in step S1 include response time, throughput, server resources, network bandwidth, software configuration and hardware configuration, CPU utilization, memory utilization, disk I / O, network traffic, network latency, radiator performance parameters, fluctuation of service response time, application performance, and network topology; the environmental data includes server temperature, data center temperature, and ambient temperature and humidity.

3. The server cluster abnormality early warning method based on fusion technology according to claim 2 is characterized in that: The performance indicators of the server cluster are monitored and collected in real time through the monitoring software installed on the server cluster. The server operation status and user access records are obtained by analyzing the log files of the server cluster. By deploying sensors in the server cluster, the server temperature and ambient temperature and humidity are monitored.

4. The server cluster abnormality early warning method based on fusion technology according to claim 1 is characterized in that: Step S2 includes the following specific processes: S21: performing data preprocessing on the collected initial feature set of the server cluster; S22: dividing the preprocessed initial feature set of the server cluster into a training set, a test set, and a validation set, which serve as training input, test input, and validation input of the feature screening model; S23: Construct a random forest model as a feature screening model to perform feature subset screening.

5. The server cluster abnormality early warning method based on fusion technology according to claim 4 is characterized in that: Step S21 includes the following specific processes: S211: De-noising: removing random noise in the initial feature set; S212: Identify abnormal data in the initial feature set, including missing values ​​and outliers; S213: Missing value and outlier processing: Processing missing value and outlier data in the initial feature set; S214: Normalization processing: converting the data in the initial feature set after noise removal into a form with a similar scale; The initial feature set of the server cluster is expressed as: ; Among them, k is the different labels or sources of the collected data, and n represents the dimension of the data, that is, the feature values ​​at different times. represents the i-th data point of the j-th type; The formula for removing noise in step S211 is: ; in, x ( i ) represents the i-th data point, t represents the position or index of the data point to be smoothed or filtered, and is an integer representing the position of a time point or data point in the time series; The missing value processing formula in step S213 is: ; in, For missing values For missing values The previous data, For missing values The latter data, is the sequence number of the data; The abnormal value processing formula in step S213 is: ; in, is the mean of x(n), is the standard deviation of x(n), Z i ( x ) is greater than the custom threshold, indicating that i Data points x ( i ) deviates too far from the mean and is marked as an outlier.

6. The server cluster abnormality early warning method based on fusion technology according to claim 4 is characterized in that: The feature screening in step S23 includes the following process: S23: constructing a random forest model, and training the random forest model using a training set; S24: Use the trained random forest model to predict the test set and calculate the importance of each feature; S25: Sort the features according to the importance of each feature; In step S24, the calculation formula of the feature importance is: ; in, I ( f ) indicates characteristics f The importance of, T represents the number of decision trees in the random forest, I t ( f ) represents the feature in the tth decision tree f The importance of I t ( f ) can be expressed by the following formula: ; Among them, among them, N t ( f ) indicates that the t-th decision tree contains the feature f The collection of nodes, N n Indicates n The number of samples of nodes, N t represents the total number of samples in the t-th decision tree, G n Indicates n The Gini index of the nodes; The formula for the Gini index is as follows: ; Among them, K represents the number of categories, Represents the proportion of samples belonging to the kth category in the nth node; The process of sorting the features in step S25 is: selecting the first m most important features as the feature subset, where m is a hyperparameter and can be adjusted according to actual conditions; the goal of selecting the feature subset is to minimize the loss function of the abnormal warning, and the loss function can be expressed by the following formula: ; Among them, P represents the accuracy of the warning, R represents the recall rate of the warning, and α and β are two weight parameters.

7. The server cluster abnormality early warning method based on fusion technology according to claim 1 is characterized in that: The anomaly detection of the fusion method in step S3 is rule-based anomaly detection, machine learning-based anomaly detection, long short-term memory network-based anomaly detection, or log and event-based anomaly detection: Rule-based anomaly detection includes setting a threshold. When the parameters of an important feature exceed the threshold, it is judged as an anomaly and triggers an anomaly warning. Machine learning-based anomaly detection includes support vector machine algorithms; Anomaly detection based on long short-term memory networks captures the long-term dependencies in feature sequence data and learns the time series patterns in the data. In the LSTM layer, multiple LSTM units are set to model the time series information and reshape the obtained features to achieve abnormal target output.

8. The server cluster abnormal warning system based on fusion technology is characterized by: It includes data acquisition module, data preprocessing module, feature screening module, fusion detection module and early warning module; The data collection module is used to collect various performance indicators and environmental data of the server cluster and construct an initial feature set; The data preprocessing module is used to preprocess the initial feature set, including cleaning, word segmentation, denoising and normalization; The feature screening module is used to select a feature subset related to the abnormal state of the server cluster from the initial feature set through a feature screening REF algorithm model; A fusion detection module, used for performing anomaly detection on the feature subset based on a fusion method; The early warning module is used to integrate the results of the detection module and output an abnormal warning report, including the type, level, location, cause, impact range, and recommended processing methods of the abnormality.

9. The server cluster abnormality early warning system based on fusion technology according to claim 8 is characterized in that: The feature screening module adopts the feature screening REF algorithm, which evaluates and sorts the initial feature set based on three indicators: relevance, redundancy and abnormality, and selects the optimal feature subset.

Citation Information

Patent Citations

  • Server exception detection method and device, server and storage medium

    CN113918438A

  • New energy combination prediction method and system based on feature extraction and optimization algorithm

    CN116542356A

  • Server fault monitoring method and system

    CN116820820A

  • Cloud server exception management method and device, computer equipment and storage medium

    CN117234844A

  • Real-time monitoring system for cloud computing server room

    CN118509248A

Cited By

  • Rail transit key equipment fault prediction method and system

    CN119397467A

  • Civil aviation freight rate anomaly detection-oriented feature screening method and system and electronic equipment

    CN119850245A

  • Intelligent server fault detection method based on BMC (Baseboard Management Controller)

    CN120849170A