Log error prediction method and electronic device based on random forest algorithm

Abstract

The embodiment of the present invention provides a log error prediction method, electronic device and readable medium based on a random forest algorithm, which relates to the field of data processing technology. The method includes: setting a sliding window; obtaining multiple load information and log information in the sliding window; constructing a time-series load feature vector according to the load information; constructing a time-series log feature vector according to the log information; setting the weights of each log feature vector; selecting some eigenvalues ​​from the log feature vector through a category-weighted roulette algorithm; splicing the eigenvalues ​​in the load feature vector with some eigenvalues ​​selected from the log feature vector to form a fused feature vector under the current sliding window; using the fused feature vectors in each sliding window as training data of the random forest algorithm to train the random forest algorithm; obtaining real-time log data; and performing error prediction on the real-time log data through the trained random forest algorithm.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a log error prediction method based on a random forest algorithm, an electronic device and a computer-readable medium. Background Art

[0002] With the rapid development of science and technology, information technology has deeply penetrated into every aspect of our lives, and various network services have emerged one after another. From social media, e-commerce, to cloud computing, the Internet of Things, and smart homes, network services play an increasingly important role in people's daily lives and work, bringing us unprecedented convenience and efficiency. However, with this comes the frequent occurrence of various service operation anomalies, which brings huge challenges to user experience and business operations. How to predict the upcoming anomalies of services has become particularly important. By early warning of service operation anomalies, corresponding measures can be taken in time to prevent the anomalies from further expanding their impact and ensure the stability and reliability of network services. Traditional monitoring and troubleshooting methods are often passive. Once an anomaly occurs, it is often too late to take measures. Therefore, the development of efficient anomaly prediction technology has become an urgent problem to be solved.

[0003] However, the prediction of service operation anomalies is a very challenging task. First, network services usually involve a large number of complex systems and huge data flows, making it very difficult to identify and locate abnormal events. The characteristics of service operation anomalies may be very complex, and different types of anomalies may show diverse forms. Traditional rules and threshold methods are difficult to adapt to various abnormal situations. Secondly, abnormal events may be diverse and uncertain, and completely new abnormal situations may appear in the future, which makes the generalization ability of prediction models face challenges. In addition, the operating status of network services usually presents a time series, and predicting service anomalies requires considering their historical changes and trends.

[0004] In order to solve these challenges, researchers continue to explore and innovate, and propose a variety of prediction methods with the help of machine learning and data mining technology. For the prediction of time series data, the strategy of multi-dimensional time window is introduced to extract features and convert time series data into a series of static samples. Compared with the traditional sliding window, the multi-dimensional sliding window can collect data of multiple dimensional indicators at the same time during a sliding process. This method can make full use of the historical information of the time series and better capture the law of service operation anomalies. Subsequently, the random forest model is trained using time series data to predict error logs. Random forest is a powerful ensemble learning algorithm that effectively reduces the risk of overfitting by constructing multiple decision trees and voting or averaging their results. With the advantages of random forest, we can more accurately capture the law of service operation anomalies, thereby improving the accuracy of prediction.

[0005] However, when using the random forest model for log error prediction, it is unable to capture the temporal order and contextual relationships of data points, and has poor ability to capture pattern trends, periodicity, or sudden changes in time series data, which leads to insufficient prediction capabilities. Summary of the invention

[0006] The embodiment of the present invention provides a log error prediction method, an electronic device and a computer-readable storage medium based on a random forest algorithm to solve the problem that when the prior art uses a random forest model for log error prediction, the time sequence and contextual relationship of data points cannot be captured, and the ability to capture pattern trends, periodic or sudden changes in time series data is poor, which leads to insufficient prediction ability.

[0007] The embodiment of the present invention discloses a log error prediction method based on a random forest algorithm, which is applied to a cloud platform and includes:

[0008] S1: Set the sliding window;

[0009] S2: Obtaining log data in the sliding window, where the log data includes multiple load information and log information;

[0010] S3: constructing a time series load feature vector at preset time intervals according to the load information;

[0011] S4: constructing a time series log feature vector according to the log information at the preset time interval;

[0012] S5: Set the weight of each log feature vector;

[0013] S6: According to the weights of the log feature vectors, select some feature values ​​from the log feature vectors through a category-weighted roulette algorithm;

[0014] S7: concatenating the eigenvalues ​​in the load eigenvector with some eigenvalues ​​selected from the log eigenvector to form a fused eigenvector under the current sliding window;

[0015] S8: using the fused feature vectors in each sliding window as training data for the random forest algorithm to train the random forest algorithm;

[0016] S9: Get real-time log data;

[0017] S10: performing error prediction on the real-time log data by using the trained random forest algorithm.

[0018] The embodiment of the present invention discloses a log error prediction device based on a random forest algorithm, which is applied to a cloud platform and includes:

[0019] A first setting module, used for setting a sliding window;

[0020] A first acquisition module, used to acquire log data in the sliding window, wherein the log data includes multiple load information and log information;

[0021] A first construction module is used to construct a time-series load feature vector at a preset time interval according to the load information;

[0022] A second construction module is used to construct a time series log feature vector according to the log information at the preset time interval;

[0023] The second setting module is used to set the weight of each log feature vector;

[0024] A selection module, used for selecting some eigenvalues ​​from the log feature vectors according to the weights of the log feature vectors through a category-weighted roulette algorithm;

[0025] A splicing module, used for splicing the eigenvalues ​​in the load eigenvector with some eigenvalues ​​selected from the log eigenvector to form a fused eigenvector under the current sliding window;

[0026] A training module is used to train the random forest algorithm by using the fused feature vectors in each sliding window as training data of the random forest algorithm;

[0027] The second acquisition module is used to acquire real-time log data;

[0028] The prediction module is used to perform error prediction on the real-time log data by using a trained random forest algorithm.

[0029] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0030] The memory is used to store computer programs;

[0031] The processor is used to implement the log error prediction method based on the random forest algorithm as described in the embodiment of the present invention when executing the program stored in the memory.

[0032] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors execute the log error prediction method based on the random forest algorithm as described in the embodiment of the present invention.

[0033] The embodiments of the present invention include the following advantages:

[0034] The present invention introduces a multi-dimensional sliding window mechanism, which retains the key feature values ​​in the most recent time period. It captures the dynamic changes of key features before an error occurs through multiple sliding operations. At the same time, it uses a category-weighted roulette algorithm to screen the most relevant feature values ​​according to the weights, thereby dynamically capturing the change patterns of features over time and the contextual relationships, and improving the log error prediction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of a log error prediction method based on a random forest algorithm provided in an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of data processing provided in an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of a characterization data structure provided in an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of a window sliding provided in an embodiment of the present invention;

[0039] Figure 5 is a block diagram of a log error prediction device based on a random forest algorithm provided in an embodiment of the present invention;

[0040] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0041] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Reference Figure 1 , shows a step flow chart of a log error prediction method based on a random forest algorithm provided in an embodiment of the present invention.

[0044] Reference Figure 2 , showing a schematic diagram of data processing provided in an embodiment of the present invention.

[0045] Reference Figure 3 , showing a schematic diagram of a characterization data structure provided in an embodiment of the present invention.

[0046] Reference Figure 4, showing a window sliding schematic diagram provided in an embodiment of the present invention.

[0047] The log error prediction method based on the random forest algorithm is applied to the cloud platform, which can specifically include the following steps:

[0048] S1: Set the sliding window.

[0049] Specifically, the width of the sliding window may be initialized.

[0050] S2: Obtain log data in the sliding window, where the log data includes multiple load information and log information.

[0051] Optionally, the load information includes: CPU load information, memory load information, network load information and disk load information.

[0052] Optionally, the log information includes: info-type log information, warn-type log information, and error-type log information. Logs at the info (information) level are mainly used to record the normal operation of the system, including program startup, stop, key execution steps, etc. This type of log information is usually used to help developers or operation and maintenance personnel understand the workflow and status of the system to ensure that the system works as expected. Logs at the warn (warning) level indicate that non-fatal but noteworthy problems have occurred in the system. Although these problems will not immediately cause system crashes or service interruptions, they may evolve into more serious failures if they persist or accumulate. Logs at the error (error) level refer to abnormal conditions that occur during system operation. These abnormalities directly hinder the normal execution of the program and may cause partial or complete functional failure.

[0053] It should be noted that a large amount of log information is generated during the operation of the server, including server hardware level information such as CPU load information, memory load information, network load information, and disk load information, and software level information such as info log information, warn log information, and error log information of various services. At present, various log information can usually be recorded in a timely manner through various means to facilitate subsequent reading and analysis. Usually, the occurrence of various service anomalies in the server does not happen overnight. There may be some signs before the anomaly occurs. By analyzing the various log information before the anomaly occurs, the occurrence of the anomaly can be predicted in some cases, reducing the impact caused by the anomaly.

[0054] It should be noted that the longer the time interval, the smaller the influence of the characteristic value related to the current error on the error. Therefore, how to choose the appropriate window interval is also particularly important. This paper proposes sliding the window to the kth window, with the maximum value of k being 5. Each time the window is slid, the sliding coefficient of the window is calculated. When the sliding coefficient is greater than 0, the characteristic values ​​of the sliding window are collected until the coefficient is less than 0.

[0055] In a possible implementation, S2 specifically includes sub-steps S201 to S203:

[0056] S201: Obtain log data in the current sliding window, and slide the current sliding window backward by one window.

[0057] S202: Calculate the sliding coefficient of the current sliding window:

[0058]

[0059] Among them, h represents the sliding coefficient, h0 represents the initial value, n0 represents the number of key eigenvalues ​​in the previous window, and n1 represents the number of key eigenvalues ​​in the current window.

[0060] It should be noted that there are a large number of low-correlation feature values ​​and some key feature values ​​in the log. Feature values ​​greater than a preset value can be determined as key feature values, and vice versa, as low-correlation feature values.

[0061] Furthermore, as the window slides, the eigenvalues ​​within the window will change. By calculating the sliding coefficient, the importance change of the current window relative to the previous window can be dynamically evaluated, so as to more flexibly respond to dynamic changes in the data stream. If the number of key eigenvalues ​​in the current window decreases, the sliding coefficient will decrease, which indicates that the current window may not be as important as the previous window, which helps to reduce the impact of the noise window on the overall analysis.

[0062] S203: Determine whether the sliding coefficient of the current sliding window is greater than or equal to 0. If so, obtain the log data in the current sliding window, and continue to slide the current sliding window backward by one window until the sliding coefficient of the current sliding window is less than 0 or has slid backward by 5 windows. Otherwise, end the data acquisition of the current sliding window.

[0063] In the present invention, when using a multi-dimensional sliding window, the situation of multiple dimensional feature values ​​in the window can be taken into account, and the feature values ​​where errors occur can be screened in many aspects when generating training data. At the same time, since the influence of the feature values ​​in the sliding window on the error when an error occurs is negatively correlated with the distance between the sliding window and the current error, the distance k of the sliding window needs to be restricted accordingly.

[0064] S3: Construct a time series load feature vector based on the load information at preset time intervals.

[0065] Among them, those skilled in the art can set the size of the preset time interval according to actual conditions, and the present invention does not limit this.

[0066] Taking CPU load information as an example, the CPU load information collected during server operation is collected at a uniform time interval t s The load moment is divided into 10 equal parts by 10% and characterized into 10 feature vectors [cpu0,cpu1,…,cpu9]. Similarly, the memory load, network load and disk load are characterized into 10 feature vectors [mem0,mem1,…,mem9], [network0,network1,…,network9] and [disk0,disk1,…,disk9] by 10 equal parts of their total amount.

[0067] S4: Based on the log information, a time series log feature vector is constructed at preset time intervals.

[0068] Among them, those skilled in the art can set the size of the preset time interval according to actual conditions, and the present invention does not limit this.

[0069] In a possible implementation, S4 specifically includes sub-steps S401 to S405:

[0070] S401: Segmenting the log information at preset time intervals.

[0071] S402: Perform word segmentation processing on the segmented log information.

[0072] It should be noted that word segmentation can split continuous log files into a single character set. However, there are a lot of useless characters in the word segmentation set, such as conjunctions, punctuation marks, numbers, proper nouns, etc. At this time, it is necessary to filter the word segmentation data to obtain a set of cleaner and more meaningful word segments.

[0073] Specifically, word segmentation can be achieved using technologies such as jieba, HanLP, THULAC, and NLTK.

[0074] S403: Filter the log information after the word segmentation processing.

[0075] Optionally, S403 specifically includes: determining whether each participle is a conjunction, punctuation mark, number or proper noun. If so, the participle is deleted. Otherwise, the participle is retained.

[0076] S404: Extract text features from the filtered log information and extract the TF-IDF features of each token.

[0077] Among them, the TF-IDF feature consists of two parts: term frequency (TF) and inverse document frequency (IDF). The term frequency refers to the frequency of a certain word appearing in the document. It reflects the importance of this word in the document. The inverse document frequency reflects the general importance of the word. The more documents a word appears in, the lower its IDF value, because it may be a common word, such as "de" (的), "shi" (是), etc.

[0078] Optionally, the TF-IDF feature is specifically:

[0079]

[0080] Among them, X j represents the TF-IDF feature value of the j-th token, TF j represents the term frequency of the j-th token, n j represents the number of times the j-th token appears in the log information, N represents the total number of tokens in the log information, m j represents the total number of documents containing the j-th token, M represents the total number of all documents, and log represents the logarithmic function with base 10.

[0081] Among them, the TF-IDF feature is a commonly used method for text feature extraction, which can quantify the importance of a word in the text, thereby converting text data into numerical features that can be used in machine learning algorithms.

[0082] S405: Construct a log feature vector according to the extracted TF-IDF features.

[0083] In the present invention, by splitting, tokenizing, filtering, and extracting TF-IDF features from the log information, the unstructured log is converted into a numerical feature vector, which can effectively remove noise, screen out key features, reduce the feature dimension, and at the same time enhance the model's ability to capture important information in the log data. This not only improves the training efficiency and prediction accuracy of the machine learning model, but also adapts to the dynamic changes of the log data, providing high-quality input data for log error prediction.

[0084] S5: Set the weights of each log feature vector.

[0085] Optionally, those skilled in the art can set the weights of each log feature vector according to the actual situation, and the present invention does not make any limitations.

[0086] Optionally, based on the experience of experts, determine which log features are most important for error prediction. For example, error logs are usually more important than warn logs, and warn logs are more important than info logs. The following weights can be set: error logs: weight 1.0, warn logs: weight 0.7, info logs: weight 0.3.

[0087] S6: According to the weights of each log feature vector, select some feature values ​​from the log feature vector through a class-weighted roulette algorithm.

[0088] Specifically, the time node t at which each error log appears x As a mark point, each time a window is advanced forward, the eigenvalues ​​in the window are used as training data for the random forest algorithm. In addition to the CPU load, memory load, disk load, and network load of the current time node, the eigenvalues ​​in the window are fixed values. Through the roulette algorithm, n values ​​are randomly selected from the eigenvalues ​​of the log file as training data for the current window. The value of n is related to the value in the current window. The window continues to advance forward until the kth window, where the value of k is related to the probability of the eigenvalue appearing and the proportion of the eigenvalue.

[0089] For example, the hardware data of the server during operation is characterized to form a time-based list [cpu0, cpu1, ..., cpu9], where 0-9 represents the current CPU load range, such as cpu1 represents the current time node CPU load of 10%-20%, similar memory, network and disk load are also characterized. s There may be multiple data in t, so the features extracted using the TF-IDF method are s There will be multiple logs in the log file. For example, the error log feature value is [(error 11 ,error 12 ,…),(error 21 ,error 22 ,…),…], where error 01Represents a feature of the current time interval, such as connection timeout, etc. Each feature corresponds to a feature value. The info and warn level logs are characterized in the same way. There are a large number of feature values ​​with low correlation and some key feature values ​​in the logs. A certain coefficient α is given to the key feature values. Feature values ​​greater than the preset value can be determined as key feature values, and vice versa. Feature values ​​with low correlation are determined. Taking the time axis of the error log as the reverse direction, the feature values ​​in the error are obtained sequentially as labels. Suppose the current time point is t0, then the CPU load, memory load, disk load and network load in the current window are cpu, respectively. i 、mem i 、disk i and network i In the log files, the impact of error logs is greater than that of warn logs and greater than that of info logs. In order to ensure the TF-IDF value of the feature value and the urgency of the feature, a roulette algorithm based on TF-IDF and feature weighting is designed to randomly select the training feature value from the log file features in the current window.

[0090] In a possible implementation, S6 specifically includes sub-steps S601 to S604:

[0091] S601: Determine the weight of each log feature value according to the weight of each log feature vector:

[0092] P ij =w i T ij

[0093] Among them, P ij represents the weight of the jth log eigenvalue in the i-th log eigenvector, w i represents the weight of the i-th log feature vector, T ij represents the jth log eigenvalue in the i-th log eigenvector.

[0094] S602: Sum the weights of all log feature values ​​to obtain a total weight.

[0095] S603: Generate a random number between 0 and the total weight.

[0096] S604: traverse each log feature value, subtract the weight of each log feature value from the random number, until the obtained random number is less than or equal to 0, and select the corresponding log feature value.

[0097] In the present invention, the random forest algorithm is used in combination with a multi-dimensional sliding window method to predict the occurrence of error logs. When using the random forest algorithm, a weighted roulette algorithm combining the TF-IDF method and the roulette method is used to screen out training data, thereby solving the problem of insufficient prediction ability of the random forest algorithm in time series data scenarios.

[0098] S7: concatenate the eigenvalues ​​in the load eigenvector with some eigenvalues ​​selected from the log eigenvector to form a fused eigenvector under the current sliding window.

[0099] Specifically, the feature values ​​of each window, including hardware data and log file data, are normalized, and the feature values ​​of each window are merged to form a fused feature vector under the current sliding window.

[0100] S8: Use the fused feature vectors in each sliding window as training data for the random forest algorithm to train the random forest algorithm.

[0101] Among them, the random forest algorithm is an ensemble learning algorithm that improves the accuracy and stability of the model by building a collection of multiple decision trees (DecisionTrees) and combining their prediction results. It is a very powerful and widely used algorithm in machine learning. The core of random forest is the decision tree. A decision tree is a tree structure in which each internal node represents a test on an attribute, each branch represents a test result, and each leaf node represents a class label or output value. Decision trees are built by recursively partitioning the data set until certain stopping conditions (such as node purity, node size, etc.) are met. Ensemble learning builds a stronger learner by combining multiple weak learners (such as a single decision tree). Random forest is a special ensemble learning method.

[0102] It should be noted that the data is imported into the random forest model for training to obtain an offline training model.

[0103] Furthermore, in the online environment, the collected operation data and operation logs are imported into the prediction model after characterization processing. If the probability of error in the model reaches the set threshold, subsequent alarm processing is performed. By manually intervening in the server operation process, the impact caused by service anomalies can be minimized.

[0104] S9: Get real-time log data.

[0105] S10: Error prediction of real-time log data using the trained random forest algorithm.

[0106] The embodiments of the present invention include the following advantages:

[0107] The present invention introduces a multi-dimensional sliding window mechanism, which retains the key feature values ​​in the most recent time period. It captures the dynamic changes of key features before an error occurs through multiple sliding operations. At the same time, it uses a category-weighted roulette algorithm to screen the most relevant feature values ​​according to the weights, thereby dynamically capturing the change patterns of features over time and the contextual relationships, and improving the log error prediction capability.

[0108] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0109] In addition, refer to Figure 5 , shows a block diagram of a log error prediction device based on a random forest algorithm provided in an embodiment of the present invention.

[0110] The embodiment of the present invention discloses a log error prediction device 20 based on a random forest algorithm, which is applied to a cloud platform and includes:

[0111] A first setting module 201, used to set a sliding window;

[0112] A first acquisition module 202, configured to acquire log data in the sliding window, wherein the log data includes multiple load information and log information;

[0113] A first construction module 203 is used to construct a time-series load feature vector at a preset time interval according to the load information;

[0114] A second construction module 204 is used to construct a time series log feature vector according to the log information at the preset time interval;

[0115] The second setting module 205 is used to set the weight of each log feature vector;

[0116] A selection module 206, configured to select some feature values ​​from the log feature vectors by using a class-weighted roulette algorithm according to the weights of the log feature vectors;

[0117] A concatenation module 207 is used to concatenate the eigenvalues ​​in the load eigenvector with some eigenvalues ​​selected from the log eigenvector to form a fused eigenvector under the current sliding window;

[0118] A training module 208 is used to train the random forest algorithm by using the fused feature vectors in each sliding window as training data of the random forest algorithm;

[0119] The second acquisition module 209 is used to acquire real-time log data;

[0120] The prediction module 210 is used to perform error prediction on the real-time log data by using a trained random forest algorithm.

[0121] In a possible implementation manner, the first acquisition module 202 is specifically configured to:

[0122] Get the log data in the current sliding window and slide the current sliding window backward by one window;

[0123] Calculate the sliding coefficient of the current sliding window:

[0124]

[0125] Where h represents the sliding coefficient, h0 represents the initial value, n0 represents the number of key eigenvalues ​​in the previous window, and n1 represents the number of key eigenvalues ​​in the current window;

[0126] Determine whether the sliding coefficient of the current sliding window is greater than or equal to 0; if so, obtain the log data in the current sliding window, and continue to slide the current sliding window backward by one window until the sliding coefficient of the current sliding window is less than 0 or has slid backward by 5 windows; otherwise, end the data acquisition of the current sliding window.

[0127] In a possible implementation manner, the load information includes: CPU load information, memory load information, network load information, and disk load information.

[0128] In a possible implementation, the log information includes: info type log information, warn type log information, and error type log information.

[0129] In a possible implementation manner, the second building module 204 is specifically configured to:

[0130] Segmenting the log information at the preset time intervals;

[0131] Perform word segmentation on the segmented log information;

[0132] Filter the log information after word segmentation processing;

[0133] Perform text feature extraction on the filtered log information and extract the TF-IDF features of each word segment;

[0134] The log feature vector is constructed according to the extracted TF-IDF features.

[0135] In a possible implementation manner, the second building module 204 is specifically configured to:

[0136] Determine whether each participle is a conjunction, punctuation mark, number or proper noun; if so, delete the participle; otherwise, retain the participle.

[0137] In a possible implementation, the TF-IDF feature is specifically:

[0138]

[0139] Among them, X j Indicates the TF-IDF feature value of the jth word, TF j represents the frequency of the jth word, n j represents the number of occurrences of the jth word in the log information, N represents the total number of words in the log information, and m j represents the total number of documents containing the jth word, M represents the total number of all documents, and log represents the logarithmic function with base 10.

[0140] In a possible implementation manner, the selection module 206 is specifically configured to:

[0141] According to the weight of each log feature vector, determine the weight of each log feature value:

[0142] P ij =w i T ij

[0143] Among them, P ij represents the weight of the jth log eigenvalue in the i-th log eigenvector, w i represents the weight of the i-th log feature vector, T ij represents the jth log eigenvalue in the i-th log eigenvector;

[0144] Sum the weights of all log eigenvalues ​​to get the total weight;

[0145] Generate a random number between 0 and the total weight;

[0146] Traverse each log feature value, subtract the weight of each log feature value from the random number, until the obtained random number is less than or equal to 0, and select the corresponding log feature value.

[0147] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The embodiments of the present invention include the following advantages:

[0149] The present invention introduces a multi-dimensional sliding window mechanism, which retains the key feature values ​​in the most recent time period. It captures the dynamic changes of key features before an error occurs through multiple sliding operations. At the same time, it uses a category-weighted roulette algorithm to screen the most relevant feature values ​​according to the weights, thereby dynamically capturing the change patterns of features over time and the contextual relationships, and improving the log error prediction capability.

[0150] In addition, refer to Figure 6 , shows a block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention also provides an electronic device, including a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304,

[0151] Memory 1303, used for storing computer programs;

[0152] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0153] S1: Set the sliding window;

[0154] S2: Obtaining log data in the sliding window, where the log data includes multiple load information and log information;

[0155] S3: constructing a time series load feature vector at preset time intervals according to the load information;

[0156] S4: constructing a time series log feature vector according to the log information at the preset time interval;

[0157] S5: Set the weight of each log feature vector;

[0158] S6: According to the weights of the log feature vectors, select some feature values ​​from the log feature vectors through a category-weighted roulette algorithm;

[0159] S7: concatenating the eigenvalues ​​in the load eigenvector with some eigenvalues ​​selected from the log eigenvector to form a fused eigenvector under the current sliding window;

[0160] S8: using the fused feature vectors in each sliding window as training data for the random forest algorithm to train the random forest algorithm;

[0161] S9: Get real-time log data;

[0162] S10: performing error prediction on the real-time log data by using the trained random forest algorithm.

[0163] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0164] The communication interface is used for communication between the above terminal and other devices.

[0165] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0166] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0167] Reference Figure 7 , showing a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. In another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored, and when the instructions are run on a computer, the computer executes the log error prediction method based on the random forest algorithm described in the above embodiment.

[0168] In another embodiment provided by the present invention, a computer program product including instructions is also provided. When the computer program product is run on a computer, the computer executes the log error prediction method based on the random forest algorithm described in the above embodiment.

[0169] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0170] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0171] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0172] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A log error prediction method based on random forest algorithm, characterized in that: include: S1: Set the sliding window; S2: Obtaining log data in the sliding window, where the log data includes multiple load information and log information; S3: constructing a time series load feature vector at preset time intervals according to the load information; S4: constructing a time series log feature vector according to the log information at the preset time interval; S5: Set the weight of each log feature vector; S6: According to the weights of the log feature vectors, select some feature values ​​from the log feature vectors through a category-weighted roulette algorithm; S7: concatenating the eigenvalues ​​in the load eigenvector with some eigenvalues ​​selected from the log eigenvector to form a fused eigenvector under the current sliding window; S8: using the fused feature vectors in each sliding window as training data for the random forest algorithm to train the random forest algorithm; S9: Get real-time log data; S10: performing error prediction on the real-time log data by using the trained random forest algorithm.

2. The log error prediction method based on random forest algorithm according to claim 1 is characterized in that: The S2 specifically includes: S201: Obtain log data in the current sliding window, and slide the current sliding window backward by one window; S202: Calculate the sliding coefficient of the current sliding window: Where h represents the sliding coefficient, h0 represents the initial value, n0 represents the number of key eigenvalues ​​in the previous window, and n1 represents the number of key eigenvalues ​​in the current window; S203: Determine whether the sliding coefficient of the current sliding window is greater than or equal to 0; if so, obtain the log data in the current sliding window, and continue to slide the current sliding window backward by one window until the sliding coefficient of the current sliding window is less than 0 or has slid backward by 5 windows; otherwise, end the data acquisition of the current sliding window.

3. The log error prediction method based on random forest algorithm according to claim 1 is characterized in that: The load information includes: CPU load information, memory load information, network load information and disk load information.

4. The log error prediction method based on random forest algorithm according to claim 1 is characterized in that: The log information includes: info type log information, warn type log information and error type log information.

5. The log error prediction method based on random forest algorithm according to claim 1 is characterized in that: The S4 specifically includes: S401: Segmenting the log information at the preset time intervals; S402: Perform word segmentation on the segmented log information; S403: Filter the log information after word segmentation processing; S404: Perform text feature extraction on the filtered log information to extract TF-IDF features of each word segment; S405: constructing the log feature vector according to the extracted TF-IDF features.

6. The log error prediction method based on random forest algorithm according to claim 5 is characterized in that: The S403 is specifically as follows: Determine whether each participle is a conjunction, punctuation mark, number or proper noun; if so, delete the participle; otherwise, retain the participle.

7. The log error prediction method based on random forest algorithm according to claim 5 is characterized in that: The TF-IDF features are specifically: Among them, X j Indicates the TF-IDF feature value of the jth word, TF j represents the frequency of the jth word, n j represents the number of occurrences of the jth word in the log information, N represents the total number of words in the log information, and m j represents the total number of documents containing the jth word, M represents the total number of all documents, and log represents the logarithmic function with base 10.

8. The log error prediction method based on random forest algorithm according to claim 1 is characterized in that: The S6 specifically includes: S601: Determine the weight of each log feature value according to the weight of each log feature vector: P ij =w i T ij Among them, P ij represents the weight of the jth log eigenvalue in the i-th log eigenvector, w i represents the weight of the i-th log feature vector, T ij represents the jth log eigenvalue in the i-th log eigenvector; S602: summing the weights of all log feature values ​​to obtain a total weight; S603: Generate a random number between 0 and the total weight; S604: traverse each log feature value, subtract the weight of each log feature value from the random number, until the obtained random number is less than or equal to 0, and select the corresponding log feature value.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the log error prediction method based on the random forest algorithm as described in any one of claims 1 to 8 when executing the program stored in the memory.

10. A computer-readable medium, characterized in that Instructions are stored thereon, which, when executed by one or more processors, enable the processors to execute the log error prediction method based on the random forest algorithm as described in any one of claims 1 to 8.

Images