Abnormal data detection method and device, equipment and storage medium

Abstract

The invention discloses an abnormal data detection method and device, equipment and a storage medium, and relates to the technical field of operation and maintaining. The method comprises the steps that a to-be-processed log is obtained, and the to-be-processed log is a data file of a character type; the to-be-processed log is sliced to obtain a first fragment set, and the first fragment set comprises a plurality of sequence fragments; the first fragment set is subjected to data transformation, a second fragment set is obtained, the second fragment set comprises a plurality of second sequence fragments, the first sequence fragments are subjected to data transformation to obtain the second sequence fragments, and the second sequence fragments are of a digital type; and clustering analysis is performed on the second fragment set to obtain abnormal data and outputting the abnormal data. According to the scheme, numeralization of the character type sequence can be achieved, clustering is automatically carried out according to probability sample distribution, hyper-parameters based on artificial experience intervention are not needed, therefore, algorithm effect differences caused by experience differences and contingency can be avoided, operability is high, and the method is easy to transplant to any new scene.

Description

technical field [0001] The present application relates to the technical field of operation and maintenance, and in particular to an abnormal data detection method, device, equipment and storage medium. Background technique [0002] Log files are often used to detect abnormal operational behavior in distributed systems. Operation and maintenance personnel often use keyword search and rule matching to manually check logs. As the scale and complexity of distributed systems continue to increase, the amount of logs increases sharply. It is inefficient to continue to rely on operation and maintenance personnel to manually check abnormalities in logs , has been unable to adapt to the increasing scale and complexity of distributed systems. [0003] At present, supervised learning and unsupervised learning are mainly used to reduce the workload of manual detection. In terms of supervised learning, there are mainly methods such as logistic regression, decision tree, and SVM. However...

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present application provides a method, device, device, and storage medium for abnormal data detection, aiming at solving the problem of relying on experience and chance in the prior art to detect abnormal operation behavior in logs based on unsupervised learning. Problems converting to numeric variables and setting hyperparameters

Images