Adaptive sampling model optimization method for data acquisition of large-scale data center

Abstract

The invention discloses an adaptive sampling model optimization method for data acquisition of a large-scale data center. The method comprises the following steps: step 1, collecting data of all limited single individuals in a data set to calculate a reconstruction error Ett so as to accurately evaluate a reconstruction result; step 2, establishing a sampling model, a reconstruction model, an estimation cost function, an error evaluation function and a loss function; step 3, dividing fully-collected data into a training set Dtrain and a test set Dtest which are independent and identically distributed; step 4, jointly training, testing and selecting the sampling model and the reconstruction model; step 5, deploying a sampling reconstruction model in real application scenario. According to the invention, the collection delay can be reduced, meanwhile, a unified collection optimization target is provided for multiple potential applications of operation data, and under the conditions that the collection cost and the reconstruction precision are comprehensively considered and all data are not observed before collection. Through establishing and optimizing the sampling model, the sampling and collecting of the operation data of the data center is adaptively done according to the incomplete historical record.

Description

technical field [0001] The invention belongs to the technical field of data collection, and in particular relates to an adaptive sampling model optimization method for large-scale data center data collection. Background technique [0002] At present, data center operation data is used for multiple data center intelligent management tasks such as energy consumption analysis and management, workflow scheduling, and task scheduling. With the increasing scale of cloud data centers, data-driven data center operation data collection become an important research question. The existing large-scale cloud data center collection methods are divided into two categories: one kind of method reduces the cost of collection by dynamically adjusting the collection strategy or collection frequency, and the other kind of method mainly uses distributed processing mechanism to collect running data. For example, the patent No. CN201310028813.7 discloses a cloud data center information difference ...

AI Technical Summary

Problems solved by technology

However, none of the above methods can realize data-driven adaptive real-time collection, increase the value density of collected data, and reduce the cost of collection tasks when the fluctuation of operating data is small, but it does not change the difficulty of collecting hundreds of thousands of nodes in large-scale data centers , because when the data fluctuates greatly, the acquisition task is still difficult to achieve real-time performance; without utilizing the inherent characteristics of the operating data, it is easy to generate a response bottleneck at the upper-level node or require a large number of data acquisition centers and processing centers. Cannot meet real-time requirements or consume too much computing resources, which is unbearable for the monitoring system

Images