A method and device for intelligent fault early warning based on indicator data
By constructing business fault prediction and indicator fault prediction models based on the K-nearest neighbor classifier, the problem of single indicator data analysis in cloud platform fault early warning is solved, and high-precision and low-load fault early warning effect is achieved.
Patent Information
- Application Number
- CN202110912497.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-10
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-08-10
AI Technical Summary
Existing technologies for fault early warning on cloud platforms suffer from limited data analysis, low prediction accuracy, high false alarm rate, excessive computational load, and difficulty in adapting to complex systems.
By extracting historical indicator data from the cloud platform, preprocessing it, and then training business fault prediction models and indicator fault prediction models, the K-nearest neighbor classifier is used for model training and prediction. Combined with LightGBM and logistic regression algorithms, a multi-dimensional fault early warning mechanism is constructed.
It improves the flexibility and accuracy of fault prediction, reduces the false alarm rate, reduces the computational load, and adapts to the fault early warning needs of complex cloud platforms.
Smart Images

Figure CN115705279B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data management technology, and specifically to an intelligent fault early warning method and device based on indicator data. Background Technology
[0002] System fault early warning refers to the rapid and accurate prediction of fault occurrence through machine learning algorithms, and the identification of the cause of the fault when it occurs, thereby reducing risk. Existing system fault early warning methods include: collecting various system indicator data, including CPU, memory, disk, and fault status; taking a portion of the data as a sample and using machine learning algorithms to train an anomaly detection model; using the remaining data to verify and adjust the model; and inputting real data into the fault early warning model to determine if there are any abnormal indicators, and issuing an alarm if so. The above-mentioned traditional fault early warning methods analyze relatively simple indicator data, do not need to consider system architecture and product characteristics, and have wide applicability, but they have the following problems: due to the overly simplistic analysis of indicator data, the prediction accuracy is low, there is a false alarm rate, which affects the judgment of operation and maintenance personnel; when applying the above-mentioned fault early warning methods to more complex cloud platforms, the training and analysis of massive indicator data places high demands on computational accuracy and speed, resulting in excessive load on the cloud platform. Summary of the Invention
[0003] In view of the above problems, the present invention is proposed to provide an intelligent fault early warning method and apparatus based on indicator data to overcome or at least partially solve the above problems.
[0004] According to one aspect of the present invention, an intelligent fault early warning method based on indicator data is provided, comprising:
[0005] Historical indicator data from the cloud platform is extracted, and the historical indicator data is preprocessed and then trained to obtain a business failure prediction model; the historical indicator data is then trained using a K-nearest neighbor classifier to obtain an indicator failure prediction model.
[0006] Obtain current indicator data from the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business.
[0007] The indicator data of the faulty service is input into the indicator fault prediction model for prediction, and the prediction result is obtained.
[0008] Determine whether the current indicator data is faulty based on the prediction results;
[0009] The historical indicator data and the current indicator data each include multi-dimensional business log data.
[0010] According to another aspect of the present invention, an intelligent fault early warning device based on indicator data is provided, comprising:
[0011] The model training module is used to extract historical indicator data from the cloud platform, preprocess the historical indicator data, and then train it to obtain a business failure prediction model; the historical indicator data is then trained using a K-nearest neighbor classifier to obtain an indicator failure prediction model.
[0012] The prediction module is used to acquire current indicator data in the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business; and input the indicator data of the failure business into the indicator failure prediction model for prediction to obtain the prediction result.
[0013] The processing module is used to determine whether the current indicator data is faulty based on the prediction results;
[0014] The historical indicator data and the current indicator data each include multi-dimensional business log data.
[0015] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;
[0016] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the intelligent fault warning method based on the above-mentioned indicator data.
[0017] According to another aspect of the present invention, a computer storage medium is provided, the storage medium storing at least one executable instruction, the executable instruction causing a processor to perform operations corresponding to the above-described intelligent fault warning method based on indicator data.
[0018] According to a method and apparatus for intelligent fault early warning based on indicator data, the present invention extracts historical indicator data from a cloud platform, preprocesses the historical indicator data, and trains it to obtain a business fault prediction model; trains the historical indicator data using a K-nearest neighbor classifier to obtain an indicator fault prediction model; acquires current indicator data from the cloud platform, inputs the current indicator data into the business fault prediction model for prediction, and outputs the indicator data of the faulty business; inputs the indicator data of the faulty business into the indicator fault prediction model for prediction, and obtains the prediction result; and determines whether the current indicator data has a fault based on the prediction result. This invention constructs a business fault prediction model using historical indicator data and an indicator fault prediction model using a K-nearest neighbor classifier. The combination of business fault prediction and indicator fault prediction enables fault early warning for the cloud platform, making fault prediction more flexible.
[0019] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0021] Figure 1 The following is a flowchart of an intelligent fault early warning method based on index data provided by an embodiment of the present invention;
[0022] Figure 2 This diagram illustrates the structure of an intelligent fault early warning device based on index data, according to an embodiment of the present invention.
[0023] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention is shown. Detailed Implementation
[0024] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
[0025] Figure 1 A flowchart illustrating an embodiment of the intelligent fault early warning method based on indicator data according to the present invention is shown, as follows: Figure 1 As shown, the method includes the following steps:
[0026] Step S110: Extract historical indicator data from the cloud platform, preprocess the historical indicator data, and train it to obtain a business failure prediction model; train the historical indicator data using a K-nearest neighbor classifier to obtain an indicator failure prediction model.
[0027] The historical and current indicator data each include multi-dimensional business log data.
[0028] In this embodiment, cloud platform product services are aggregated, including cloud computing, storage, database, middleware, and other product services. Historical data related to the above-mentioned cloud platform product services are collected as historical indicator data. In addition, historical indicator data also includes other hardware data of the cloud platform, such as host performance data, alarm data, and fault log data (e.g., fault handling logs of physical machines / virtual machines).
[0029] In an optional approach, step S110 further includes: normalizing historical indicator data according to preset rules to obtain a historical dataset; sampling the historical dataset, extracting features from the sampled data to obtain first feature data, inputting the first feature data into the LightGBM model to obtain second feature data; and training the first feature data and the second feature data using a logistic regression algorithm model to obtain a business failure prediction model.
[0030] Specifically, historical indicator data from the previous month or week can be collected as needed. Preprocessing of the historical indicator data involves observing the data to determine how to handle missing historical indicator data (e.g., retaining or discarding it). Generally, data useful for building the fault prediction model should be retained. Then, the historical indicator data is normalized to form a unified format historical dataset for easy input into the model for training.
[0031] Furthermore, data sampling is performed on the historical dataset, and feature extraction is performed on the sampled data to obtain the first feature data. The first feature data is then input into the Light Gradient Boosting Machine (LightGBM) model to obtain the second feature data. Specifically, in order to improve the efficiency of machine model learning during model construction without sacrificing the model's prediction accuracy, histogram algorithm and Gaussian (GOSS) algorithm are used to classify and sample the historical dataset. Classification sampling means classifying the historical indicator data according to the gradient distribution and setting different sampling rates and weights. At the same time, the mutually exclusive feature binding (EFB) algorithm is used to reduce the dimensionality of the historical dataset, and each dimension of the collected business log data is processed into a first feature data. The LightGBM model uses a histogram algorithm to find the optimal split point. It discretizes the feature values of each sampled data point using a binning method, assigning values within a certain range to specific bins, thus discretizing continuous floating-point features into k discrete values. Then, it constructs a histogram of width k to replace the original data. Finally, it uses the constructed histogram to traverse the data, calculating the gradient and sample size in each bin to find the optimal split point, eliminating the need to traverse all data individually, significantly reducing computation and improving training speed. The first feature data is input into the LightGBM model to obtain the second feature data. The first and second feature data are then trained using a Logistic Regression (LR) model. The LR algorithm maps the result of a linear function to a sigmoid function to predict and classify the probability of fault events, thus it can be used to study the influence relationship between feature data (performance index data) and whether a fault occurs.
[0032] It is important to note that a one-sided gradient sampling strategy is employed when sampling historical datasets. Since each historical data instance has a different gradient, according to the definition of information gain, instances with larger gradients have a greater impact on information gain. Therefore, during sampling, samples with large gradients are retained (specifically, this can be determined by pre-setting a threshold or the highest percentile), while samples with small gradients are randomly removed. Information gain is calculated using only the remaining samples, which better simulates the distribution of historical data and improves the generalization ability of the business fault early warning model. The EFB algorithm is used to reduce the dimensionality of the historical dataset. When collecting feature data, the features of each dimension of the collected business log data need to be sorted according to the number of non-zero values. The conflict ratio between different features is calculated, and mutually exclusive features such as one feature value being zero and another not being zero are bundled into low-dimensional, dense features. This effectively avoids the calculation of unnecessary zero-value features, thereby improving computational efficiency.
[0033] First, LightGBM is used to train the constructed first feature data to construct new feature data, namely the second feature data, to obtain a binary classifier. Grid search is used to find the optimal parameter combination. In this embodiment, the loss function used in the binary classification problem is the log loss function. When constructing the second feature data using the trained LightGBM model, the value of the second feature data feature vector is 0 / 1. Each element of the feature vector corresponds to a leaf node of the tree in the LightGBM model. When the tree learned by the LightGBM model makes a prediction, if a sample point eventually falls on a leaf node of a certain tree, then the element corresponding to this leaf node in the new feature vector is 1, while the element corresponding to the other leaf nodes of this tree is 0. The length of the feature vector of the second feature data is equal to the sum of the number of leaf nodes contained in all trees in the GBDT model. The first and second feature data are input into the LR algorithm model for training the final classifier. The loss function of the LR algorithm model is the following formula (1):
[0034]
[0035] Among them, y i x represents the label corresponding to sample i. i h represents the value of sample i. θ (x i ) represents the probability of sample i, and m represents the number of samples.
[0036] The trained business failure prediction model is used to predict the current indicator data collected to determine whether a failure has occurred.
[0037] In one optional approach, the historical indicator data includes first historical indicator data for normal operations and second historical indicator data for faulty operations. The business label of the first historical indicator data is the first label, and the business label of the second historical indicator data is the second label. Step S110 further includes: training the first historical indicator data and the second historical indicator data using a K-nearest neighbor classifier to obtain the distribution of each historical indicator data in the normal category and the faulty category, as well as the Euclidean distance between any two historical indicator data belonging to the same category; adjusting the parameters of the K-nearest neighbor classifier based on the distribution, the Euclidean distance, and the business label corresponding to each historical indicator data to obtain the trained indicator fault prediction model.
[0038] In this embodiment, the stacking concept is used to fuse the LightGBM model and the LR algorithm model, transforming the prediction problem into a classification problem. Historical indicator data is classified into first historical indicator data for normal operations and second historical indicator data for faulty operations. The business label for the first historical indicator data is the first label, and the business label for the second historical indicator data is the second label. Further, the first and second historical indicator data are trained using a K-Nearest Neighbor (KNN) classifier to obtain the distribution of each historical indicator data in the normal and faulty categories, as well as the Euclidean distance between any two historical indicator data belonging to the same category. Specifically, the basic idea of the Stacking model is as follows: assuming there are 1000 training data points and 100 test data points, the training set is divided into 5 parts (generally 5 parts), with 200 data points in each part. The model is trained on four of these parts, i.e., 800 data points. Then, the remaining 200 data points are predicted, and the 100 test data points are also predicted to obtain the prediction results. After 5 training iterations, the training set yields exactly 200×5 results, which is the original number of results in the training set. These are combined into a single column, resulting in a 1000×1 matrix. The test set yields 100×5 results. The average of the 5 predictions is then taken, resulting in a 100×1 matrix, completing the first layer task. Next, the same method is used to try other models. The results from different models are combined column-wise. If three base models are used, a 1000×3 matrix and a 100×3 matrix are obtained. These results are used as the training and test sets for the second layer model, with the initial training set labels used as the second layer training set labels. The model is then used for training and prediction.
[0039] The Euclidean distance between any two historical index data belonging to the same category is calculated using the following formula (2):
[0040]
[0041] Where x and y represent the values of two different historical indicator data belonging to the same category (normal category or fault category); d(x,y) represents the distance between the two values.
[0042] The parameters of the K-nearest neighbor classifier are adjusted based on the distribution, Euclidean distance, and business labels corresponding to each historical indicator data to obtain the trained indicator fault prediction model.
[0043] Step S120: Obtain the current indicator data in the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business; input the indicator data of the failure business into the indicator failure prediction model for prediction and obtain the prediction result.
[0044] In an optional manner, step S120 further includes: inputting the indicator data of the faulty service into the indicator fault prediction model for prediction, calculating the Euclidean distance between the indicator data of the faulty service and each historical indicator data; and determining the prediction result based on the service label of the historical indicator data with the smallest Euclidean distance.
[0045] Step S130: Determine whether the current indicator data is faulty based on the prediction results.
[0046] In an optional approach, step S130 further includes: if the business label in the prediction result is the label corresponding to a normal business, then it is determined that the current indicator data has not failed; if the business label in the prediction result is the label corresponding to a faulty business, then it is determined that the current indicator data has failed.
[0047] Using the method of this embodiment, a business failure prediction model and an indicator failure prediction model are obtained by training a batch of MySQL products through steps S110-S130. Table 1 shows an example of the collected historical indicator data samples. As shown in Table 1, X1-X17 are 17 indicators of the cloud platform MySQL product business. Each column is the indicator data, and each row is the value of the indicator data. X1-X17 include: CPU utilization (%), CPU load (average), memory utilization (%), disk space utilization (%), disk IOPS (times per second), disk throughput (MB / s), and network speed. The following metrics were used to construct sample data: (Mb / s), bandwidth utilization (%), number of system processes, CPU temperature (°C), fan speed (rpm), MySQL query throughput (number of queries per second), MySQL persistent connection utilization (%), MySQL query cache space utilization (%), MySQL query cache hit rate (%), MySQL cache queries, and MySQL index cache hit rate (%). This includes the first historical indicator data for normal business operations and the second historical indicator data for faulty business operations. The data is shown in Table 1, where a label of 0 indicates normal business operations and 1 indicates faulty business operations.
[0048] X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 label 0.08 0.4 0.53 0.75 1000 1 3 0.3 280 50 2300 5000 0.33 0.34 0.15 200000 0.5 0 0.4 0.8 0.77 0.5 5 50 4 0.5 2500 70 3500 8000 0.65 0.67 0.37 333456 0.69 0 … … … … … … … … … … 0.99 0.99 0.9 0.88 10000 500 9 3.2 5000 80 5000 20000 0.9 0.9 0.7 555213 0.89 1
[0049] Table 1 Example of Historical Indicator Data Sample
[0050] The sample data in Table 1 above is trained using the LightGBM model. Specifically, the sample data is converted into a sparse feature vector by transforming the elements corresponding to the leaf nodes in each tree of the trained LightGBM model into elements with a value of 1, while the elements corresponding to the other leaf nodes in that tree are 0. The trained LightGBM model has n weak classifiers and m leaf nodes. Therefore, each first feature data point is converted into a 1*m dimensional sparse vector, with n elements being 1 and the remaining mn elements being 0. All zeros, the data is converted to the form [0.0.0....1.0.0.], which is the second feature data; the second feature data in the form [0.0.0....1.0.0.] is fed into the LR algorithm model, and the probability corresponding to the label [0, 1] for each sample data can be calculated. The closer the value is to 1, the greater the possibility of the cause of the fault; the array format is as follows: [0.04403279, 0.95596721]. The two values in the array are compared with 1 to determine whether they are close to 1, thereby obtaining the information of whether there is a fault.
[0051] Furthermore, the fault indicators predicted by the business fault prediction model are used to predict fault indicators by calculating Euclidean distance using the trained indicator fault prediction model. Based on the prediction results, it is determined whether the current indicator data has a fault, so as to remind the operation and maintenance personnel to perform key maintenance.
[0052] Taking the CPU utilization (%) mentioned above as an example: In the training of KNN, the business label of the CPU utilization sample data obtained by training on historical fault data of CPU utilization is represented as [Euclidean distance, 'probability of fault cause']. The data obtained from the training is as follows:
[0053] {"cpu utilization 1":[0.45, '0'],"cpu utilization 2":[0.21, '0'],"cpu utilization 3":[0.54, '0'],"cpu utilization 4":[0.39, '0'],"cpu utilization 5":[0.5, '0'],"cpu utilization 6":[0.3, '0'],"cpu utilization 7":[0.2, '0'],"cpu utilization 8":[0.6, '0'],"cpu utilization 9":[0.7, '0'],"cpu utilization 10":[0.49, '0'],"cpu utilization 11":[0.38, '0'],"cpu utilization 12":[0.17, '0'],"cpu utilization CPU utilization 13":[0.88, '1'], "CPU utilization 14":[0.89, '1'], "CPU utilization 15":[0.9, '1'], "CPU utilization 16":[0.91, '1'], "CPU utilization 17":[0.92, '1'], "CPU utilization 18":[0.93, '1'], "CPU utilization 19":[0.94, '1'], "CPU utilization 20":[0.95, '1'], "CPU utilization 21":[0.96, '1'], "CPU utilization 22":[0.97, '1'], "CPU utilization 23":[0.98, '1'], "CPU utilization 24":[0.99, '1']}
[0054] One of the business data predicted by the business failure prediction model has a CPU utilization rate of [0.9]. Substituting this data into the trained indicator failure prediction model, the Euclidean distance between this indicator data and the KNN training data results is calculated as follows:
[0055] ['CPU utilization 22', 0.07];
[0056] Here, 22 refers to the 22nd record of the training data results of the above indicator fault prediction model; 0.07 is the Euclidean distance between the business data [0.9] and "cpu utilization 17": [0.92, '1'].
[0057] The faulty service metrics data are input into the metric fault prediction model for prediction, resulting in the following data:
[0058] ['CPU utilization 6', 0.6], ['CPU utilization 21', 0.06], ['CPU utilization 4', 0.51], ['CPU utilization 19', 0.04], ['CPU utilization 20', 0.05], ['CPU utilization 17', 0.02], ['CPU utilization 18', 0.03], ['CPU utilization 15', 0.0], ['CPU utilization 13', 0.02], ['CPU utilization 9', 0.2], ['CPU utilization 1', 0.45], ['CPU utilization... [5', 0.4], ['cpu utilization 23', 0.08], ['cpu utilization 8', 0.3], ['cpu utilization 7', 0.7], ['cpu utilization 12', 0.73], ['cpu utilization 16', 0.01], ['cpu utilization 2', 0.69], ['cpu utilization 24', 0.09], ['cpu utilization 10', 0.41], ['cpu utilization 14', 0.01], ['cpu utilization 11', 0.52], ['cpu utilization 3', 0.36].
[0059] Applying the above faulty service indicator data [0.9] to the trained indicator fault prediction model, the shortest Euclidean distance is found in the 15th item, which is 0 and predicts a fault. Therefore, the CPU utilization indicator data is close to the fault category of category 1, indicating that the current indicator data [0.9] has a fault, and maintenance personnel can be notified to handle it with priority.
[0060] This invention utilizes the stacking concept to fuse the LightGBM model and the LR algorithm to train historical indicator data, transforming the prediction problem into a classification problem, thus constructing a business fault prediction model. The K-nearest neighbor classifier is then used to construct an indicator fault prediction model. This combination of business fault prediction and indicator fault prediction enables more flexible fault prediction. Compared to independent prediction models, it offers stronger nonlinear representation capabilities, reduces generalization error, minimizes overfitting, and improves the accuracy of model prediction and classification.
[0061] Figure 2 A schematic diagram of an embodiment of the intelligent fault early warning device based on indicator data according to the present invention is shown. Figure 2 As shown, the device includes: a model training module 210, a prediction module 220, and a processing module 230.
[0062] The model training module 210 is used to extract historical indicator data from the cloud platform, preprocess the historical indicator data, and then train it to obtain a business failure prediction model; the historical indicator data is trained using a K nearest neighbor classifier to obtain an indicator failure prediction model.
[0063] In one alternative approach, historical metric data and current metric data each include multi-dimensional business log data.
[0064] In an optional manner, the model training module 210 is further configured to: normalize historical indicator data according to preset rules to obtain a historical dataset; sample the historical dataset, extract features from the sampled data to obtain first feature data, input the first feature data into the LightGBM model to obtain second feature data; and train the first feature data and the second feature data using a logistic regression algorithm model to obtain a business fault prediction model.
[0065] In one optional approach, the historical indicator data includes first historical indicator data for normal operations and second historical indicator data for faulty operations. The business label of the first historical indicator data is the first label, and the business label of the second historical indicator data is the second label. The model training module 210 is further used to: train the first historical indicator data and the second historical indicator data through a K-nearest neighbor classifier to obtain the distribution of each historical indicator data in the normal category and the faulty category, as well as the Euclidean distance between any two historical indicator data belonging to the same category; adjust the parameters of the K-nearest neighbor classifier according to the distribution, the Euclidean distance, and the business label corresponding to each historical indicator data to obtain the trained indicator fault prediction model.
[0066] The prediction module 220 is used to obtain the current indicator data in the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business; input the indicator data of the failure business into the indicator failure prediction model for prediction and obtain the prediction result.
[0067] In an alternative approach, the prediction module 220 is further configured to: input the indicator data of the faulty service into the indicator fault prediction model for prediction, calculate the Euclidean distance between the indicator data of the faulty service and each historical indicator data, and determine the prediction result based on the service label of the historical indicator data with the smallest Euclidean distance.
[0068] The processing module 230 is used to determine whether the current indicator data has malfunctioned based on the prediction results.
[0069] In an optional manner, the processing module 230 is further configured to: if the business label in the prediction result is the label corresponding to a normal business, then determine that the current indicator data has not failed; if the business label in the prediction result is the label corresponding to a faulty business, then determine that the current indicator data has failed.
[0070] The apparatus of this embodiment extracts historical indicator data from the cloud platform, preprocesses the historical indicator data, and trains it to obtain a business fault prediction model; it then trains the historical indicator data using a K-nearest neighbor classifier to obtain an indicator fault prediction model; it acquires current indicator data from the cloud platform, inputs the current indicator data into the business fault prediction model for prediction, and outputs the indicator data of the faulty business; it then inputs the indicator data of the faulty business into the indicator fault prediction model for prediction, and obtains the prediction result; finally, it determines whether the current indicator data has experienced a fault based on the prediction result. This apparatus constructs a business fault prediction model using historical indicator data and an indicator fault prediction model using a K-nearest neighbor classifier. The combination of business fault prediction and indicator fault prediction enables fault early warning for the cloud platform, making fault prediction more flexible.
[0071] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute an intelligent fault early warning method based on indicator data in any of the above method embodiments.
[0072] Executable instructions can specifically be used to cause the processor to perform the following operations:
[0073] Historical indicator data from the cloud platform is extracted, preprocessed, and then used to train a business failure prediction model; the historical indicator data is then used to train an indicator failure prediction model using a K-nearest neighbor classifier.
[0074] Obtain the current indicator data from the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business.
[0075] Input the indicator data of the faulty business into the indicator fault prediction model for prediction, and obtain the prediction result;
[0076] Determine whether the current indicator data is faulty based on the forecast results;
[0077] The historical and current indicator data each include multi-dimensional business log data.
[0078] Figure 3 The diagram shows a structural schematic of an embodiment of the computing device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.
[0079] like Figure 3 As shown, the computing device may include:
[0080] Processor, Communications Interface, Memory, and Communications Bus.
[0081] The processor, communication interface, and memory communicate with each other via a communication bus. The communication interface is used to communicate with other network elements, such as clients or other servers. The processor executes programs, specifically the relevant steps in the above-described embodiment of an intelligent fault early warning method based on indicator data.
[0082] Specifically, the program may include program code, which includes computer operation instructions.
[0083] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The server may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.
[0084] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.
[0085] Specifically, the program can be used to cause the processor to perform the following operations:
[0086] Historical indicator data from the cloud platform is extracted, preprocessed, and then used to train a business failure prediction model; the historical indicator data is then used to train an indicator failure prediction model using a K-nearest neighbor classifier.
[0087] Obtain the current indicator data from the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business.
[0088] Input the indicator data of the faulty business into the indicator fault prediction model for prediction, and obtain the prediction result;
[0089] Determine whether the current indicator data is faulty based on the forecast results;
[0090] The historical and current indicator data each include multi-dimensional business log data.
[0091] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of the invention.
[0092] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0093] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.
[0094] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.
[0095] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.
[0096] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.
[0097] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.
Claims
1. An intelligent failure warning method based on index data, characterized by, include: Historical indicator data of the cloud platform is extracted, and the historical indicator data is preprocessed and then trained to obtain a business failure prediction model; the historical indicator data is then trained using a K-nearest neighbor classifier to obtain the indicator failure prediction model; wherein, the historical indicator data includes historical data related to cloud platform product services and other hardware data of the cloud platform, the other hardware data includes host performance data, alarm data and fault log data; the cloud platform product services include cloud computing, storage, database and middleware components; Obtain current indicator data from the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business. The indicator data of the faulty service is input into the indicator fault prediction model for prediction. The Euclidean distance between the indicator data of the faulty service and each historical indicator data is calculated. The prediction result is obtained based on the service label of the historical indicator data with the smallest Euclidean distance. Determine whether the current indicator data is faulty based on the prediction results; The historical indicator data and the current indicator data each include multi-dimensional business log data.
2. The method of claim 1, wherein, The step of preprocessing the historical indicator data and training it to obtain the business failure prediction model further includes: The historical indicator data is normalized according to preset rules to obtain a historical dataset. Data sampling is performed on the historical dataset, and feature extraction is performed on the sampled data to obtain first feature data. The first feature data is then input into the LightGBM model to obtain second feature data. The first feature data and the second feature data are trained using a logistic regression algorithm model to obtain a business failure prediction model.
3. The method of claim 1, wherein, in, The historical indicator data includes first historical indicator data for normal business and second historical indicator data for faulty business. The business label of the first historical indicator data is the first label, and the business label of the second historical indicator data is the second label. The step of training the historical indicator data using a K-nearest neighbor classifier to obtain the indicator fault prediction model further includes: The first historical indicator data and the second historical indicator data are trained using a K-nearest neighbor classifier to obtain the distribution of each historical indicator data in the normal category and the fault category, as well as the Euclidean distance between any two historical indicator data belonging to the same category. Based on the distribution, the Euclidean distance, and the business labels corresponding to each historical indicator data, the parameters of the K nearest neighbor classifier are adjusted to obtain the trained indicator fault prediction model.
4. The method according to claim 1, characterized in that, The step of determining whether the current indicator data is faulty based on the prediction result further includes: If the business label in the prediction result is the label corresponding to normal business, then it is determined that the current indicator data has not malfunctioned; If the business label in the prediction result is the label corresponding to the faulty business, then it is determined that the current indicator data is faulty.
5. An intelligent fault early warning device based on indicator data, characterized in that, include: The model training module is used to extract historical indicator data from the cloud platform, preprocess the historical indicator data, and then train it to obtain a business failure prediction model; the historical indicator data is then trained using a K-nearest neighbor classifier to obtain the indicator failure prediction model; wherein, the historical indicator data includes historical data related to cloud platform product services and other hardware data of the cloud platform, the other hardware data including host performance data, alarm data, and fault log data; the cloud platform product services include cloud computing, storage, database, and middleware components; The prediction module is used to obtain current indicator data in the cloud platform, input the current indicator data into the business failure prediction model for prediction, and output the indicator data of the failure business; input the indicator data of the failure business into the indicator failure prediction model for prediction, calculate the Euclidean distance between the indicator data of the failure business and each historical indicator data, and obtain the prediction result based on the business label of the historical indicator data with the smallest Euclidean distance. The processing module is used to determine whether the current indicator data is faulty based on the prediction results; The historical indicator data and the current indicator data each include multi-dimensional business log data.
6. The apparatus according to claim 5, characterized in that, The model training module is further used for: The historical indicator data is normalized according to preset rules to obtain a historical dataset. Data sampling is performed on the historical dataset, and feature extraction is performed on the sampled data to obtain first feature data. The first feature data is then input into the LightGBM model to obtain second feature data. The first feature data and the second feature data are trained using a logistic regression algorithm model to obtain a business failure prediction model.
7. The apparatus according to claim 5, characterized in that, in, The historical indicator data includes first historical indicator data for normal business and second historical indicator data for faulty business. The business label of the first historical indicator data is the first label, and the business label of the second historical indicator data is the second label. The model training module is further used for: The first historical indicator data and the second historical indicator data are trained using a K-nearest neighbor classifier to obtain the distribution of each historical indicator data in the normal category and the fault category, as well as the Euclidean distance between any two historical indicator data belonging to the same category. Based on the distribution, the Euclidean distance, and the business labels corresponding to each historical indicator data, the parameters of the K nearest neighbor classifier are adjusted to obtain the trained indicator fault prediction model.
8. A computing device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the intelligent fault early warning method based on indicator data as described in any one of claims 1-4.
9. A computer storage medium, characterized in that, The storage medium stores at least one executable instruction, which causes the processor to perform the operation corresponding to the intelligent fault early warning method based on indicator data as described in any one of claims 1-4.
Citation Information
Patent Citations
Social network link anomaly detection method utilizing cascading generalization and cost-sensitive learning
CN112073227A
