Stream processing method and system for online filling of network missing data

Abstract

The invention discloses a stream processing method for online filling of network missing data, which comprises the following steps: in the process of performing stream processing on the network missing data, extracting time and space information contained in a monitoring data matrix sequence in a previous time period through a feature extractor and a gating circulation unit; a context vector which retains effective information in historical data can be obtained, the context vector is combined with spatial feature vector information corresponding to network missing data at the current moment, and a combined vector is input into a pre-trained missing data generation model so as to obtain a current monitoring data matrix filled with the network missing data. According to the stream processing method and system for online filling of the network missing data provided by the invention, the spatial feature vector corresponding to the network missing data is combined with the context vector related to the previous historical data, so that the spatial-temporal information of the network data in the previous time period is fused, and the precision of online filling of the network missing data can be effectively improved.

Description

technical field [0001] The invention belongs to the technical field of computer networks, and more specifically relates to a stream processing method and system for online filling of network missing data. Background technique [0002] With the rapid development of communication technology, the scale of the network that needs to be maintained is also increasing. In the process of network operation and maintenance, it is necessary to measure network monitoring data for subsequent tasks such as anomaly detection, root cause analysis, and traffic prediction. However, the cost of performing network-wide measurement on a large-scale network is very high, and frequent measurement will also occupy a large amount of network resources. [0003] Studies have shown that network monitoring data has temporal correlation and spatial correlation, so the current mainstream processing method is to measure part of the data and restore the unmeasured data according to the correlation. Since t...

AI Technical Summary

Problems solved by technology

[0007] In view of the above defects or improvement needs of the prior art, the present invention provides a stream processing method and system for online filling of missing data in the network. The matrix is ​​used as input, and each sequence needs to train parameters from scratch, which leads to the technical problem of greatly increasing the time cost of filling, and the existing machine learning-based filling method does not train a filling model with fixed parameters through historical data. It is applicable to the technical problems of filling network data, as well as the technical problems of slow convergence speed of the model due to failure to consider the characteristics of network data conforming to the heavy-tailed distribution, and the defect of unbalanced gradient update in the process of training the model, resulting in the final The technical problem of low filling accuracy

Images