Data acquisition speed control method and device and storage medium

Abstract

The invention discloses a data collection speed control method and device and a storage medium, and the method comprises the steps: collecting data of a data source, and requesting to write the data into a system temporary storage; and adjusting the sleep time of the system between two times of data acquisition according to a judgment result of whether data writing is successful or not and the usecapacity of the temporary storage. The equipment and the storage medium realize the steps of the data acquisition speed control method. Different dormancy strategies are adopted according to different states such as data writing and the use capacity of the temporary storage, so that the whole data acquisition process can be stably carried out, and the impact on the stability of the whole acquisition process due to sudden increase or sudden decrease of the acquisition frequency is avoided.

Description

technical field [0001] The invention relates to the field of data acquisition, in particular to a data acquisition speed control method, device and storage medium. Background technique [0002] In the era of big data, data collection tools such as Flume have become the main way of data collection. Flume is a distributed, reliable, and highly available mass log collection, aggregation and transmission system. As an excellent data collection tool, Flume can be applied to multiple scenarios. The Agent process is an independent process, including Source, Channel, and Sink components. Source is the component that acquires data in Flume and writes data to In the Channel, the Channel is a passive memory, responsible for temporarily storing the data sent by the Source, and the Sink is responsible for consuming the data in the Channel and sending it to the corresponding destination. figure 1 For Flume's data collection architecture, Source can collect data from log files, network p...

AI Technical Summary

Problems solved by technology

[0003] In the existing data writing process, due to reasons such as network failure, high host load, and slow processing speed of the destination component, the Sink will write data slowly or fail to write data for a period of time, while the Source is collecting The speed of data will not change because of this, resulting in a large backlog of data in the Channel

When the data stored in the Channel reaches the maximum value set, the Source will not be able to write data to the Channel, so it will continue to retry according to a fixed time, occupying the computing resources of the host and increasing the load of the host

Moreover, due to the backlog of a large amount of data in the Channel, it will also occupy a large amount of storage resources and cannot be released

Images