Cluster log feature extraction method and device and storage medium

Abstract

The invention relates to base frame operation and maintenance, and provides a cluster log feature extraction method and device and a storage medium, and the method comprises the steps: collecting a log of a server cluster through a flume client, and sending the log to a database; performing data cleaning on the log data, and screening out original data; extracting characteristic values including amean value, an effective value, a peak value, a square root amplitude value, a waveform index, a pulse index and a kurtosis index from the original data; and respectively carrying out Pearson correlation coefficient operation on the extracted characteristic values and original data, comparing the calculated correlation coefficient with a correlation degree threshold value, regarding the data as valid data if the correlation degree is higher than the correlation degree threshold value, regarding the data as invalid data if the correlation degree is lower than the correlation degree threshold value, and eliminating the invalid data. According to the invention, the effective information of the production data of each host in the server cluster can be effectively screened out, and the characteristic values of the production data are extracted from the effective information, so that the fault prediction and fault classification of a production system are facilitated, and the occurrence ofproduction accidents is reduced.

Description

technical field [0001] The present invention relates to base frame operation and maintenance, in particular to a cluster log feature extraction method, device and storage medium. Background technique [0002] In the era of explosive growth of information, the file size and data scale have become a reality, and the number of nodes in the cluster storage system has reached 64 node clusters. Managing such a large cluster system has become a severe challenge for the data center. challenge. It is particularly important to track the running status of cluster nodes in time and accurately locate node error information. In the actual operation of the cluster storage system, a cluster storage system log management method is commonly used at present, which can send system logs regularly or in real time, realizing the centralized transmission of logs, but without analyzing and managing the logs, it is impossible to understand the entire cluster globally The running status of the stora...

AI Technical Summary

Problems solved by technology

In the actual operation of the cluster storage system, a cluster storage system log management method is commonly used at present, which can send system logs regularly or in real time, and realize the centralized transmission of logs, but without analyzing and managing the logs, it is impossible to understand the entire cluster globally The operation status of the storage system cannot quickly locate the error message

However, as the number of cluster nodes increases, the management of the cluster system becomes more and more complicated

Images