An unsupervised learning method-based abnormal monitoring scheme of a stream computing system

Abstract

The invention provides an abnormity monitoring scheme of a flow computing system based on an unsupervised learning method, belongs to the field of distributed real-time system abnormity monitoring, and specifically comprises a system behavior description module, an abnormity monitoring module constructed based on a small sample constraint condition, and an online adaptive abnormity monitoring module. The method comprises the following steps: firstly, converting an original event by utilizing an event processing technology to obtain a composite event; therefore, the event state data index and the physical state data index are obtained, the collected data indexes are fused through the time window technology to obtain the system behavior state index space, and behavior description of the stream computing system is achieved. Secondly, proposing an unsupervised statistical analysis method, constructing an exception monitoring model based on a small sample constraint condition, and realizingexception monitoring of unbalanced data of the stream computing system; and finally, providing an on-line adaptive abnormity monitoring model, automatically adjusting a network structure, updating aclustering center, and realizing on-line adaptive abnormity monitoring.

Description

technical field [0001] The invention belongs to the field of abnormal monitoring of distributed real-time systems, and mainly relates to the establishment of online monitoring models of distributed real-time systems, in particular to a multi-abnormal clustering method based on system behavior states. Background technique [0002] With the continuous expansion of the scale of the big data industry and the continuous expansion of the field of big data applications, stream computing systems, as an emerging big data processing mode, have gradually become an important tool for people's production and application. In large-scale production applications, thousands of nodes process data online in real time at the same time, which makes the security of the flow computing system a focus of attention, and its security accidents will bring huge economic losses and immeasurable intangible losses to the society. , therefore, the credibility, reliability, and security of stream computing s...

AI Technical Summary

Problems solved by technology

Anomaly monitoring based on semi-supervised learning methods is an improvement over supervised methods, using less training sets to train classifiers, but its disadvantage is that it is not sensitive to novel classes or unknown anomalies, such as the literature "A Novel Transductive SVM for S3VM proposed by theSemisupervised Classification of Remote-Sensing Images

Unsupervised learning does not need to label data, overcomes the shortcomings of supervised learning, and becomes a new direction for anomaly monitoring of stream computing systems. For example, the document "Self-adaptive and dynamic clustering for online anomaly detection" proposes using Kmeans for anomaly monitoring. At the same time, this paper introduces the neural network algorithm SOM to solve the problem of unsatisfactory clustering effect of unbalanced samples, but the topology of the neural network is unstable, and the quality of its abnormal monitoring largely depends on the weight setting.

Images