A Distributed File System Write Access Method for Parallel Data Acquisition

Abstract

The invention discloses a parallel data acquisition-oriented write access method for a distributed file system. A metadata processing process in a synchronous running state is consistent with a conventional method; a metadata processing process in a mode discovery running state is consistent with the synchronous running state, but all metadata requests and processing results in the state need to be recorded, and statistic analysis is performed according to a recording result; and in an asynchronous running state, a client, according to a storage record, predicts a metadata processing result and directly returns a prediction result to a data acquisition program, then the metadata requests are sent to a metadata server for processing, and if the processing result is consistent with the prediction result, no additional processing is performed, or otherwise, an exception processing mode is immediately performed. The method has the following advantages: the network delay when the data acquisition program waits for the metadata processing result can be greatly shortened, so that the write performance of the system is further improved.

Description

technical field [0001] The invention relates to the field of distributed storage, in particular to a write access method of a distributed file system oriented to parallel data collection. Background technique [0002] With the promotion of smart cities, various parallel data acquisition devices have been widely used in people's lives, such as cameras in traffic networks. Such devices are usually installed in every corner of the city, use hardware to collect data, and store the collected data in a distributed file system, allowing users to easily read relevant data from the file system. A distributed file system usually includes three parts: client, server, and metadata server. The client is deployed in the computer where the application program is located to provide users with data read and write interfaces; the data server is usually deployed in the server in the data center for Store file content; the metadata server is also usually deployed in the server of the data cent...

AI Technical Summary

Problems solved by technology

[0003] At present, due to the upgrade of various data acquisition devices, the amount of data collected by such devices becomes larger and larger every second. For example, high-definition cameras generate a large amount of data every second, while the traditional distributed file system has It is difficult to meet the I / O requirements of these devices; one solution to this problem is to integrate data acquisition devices and corresponding data acquisition programs into the distributed file system, supplemented by special physical layouts and special algorithms and data structures , resulting in a distributed file system oriented to parallel data acquisition, which satisfies the bandwidth requirements of data acquisition programs to a certain extent

However, in the actual processing process, since the metadata processing process of the distributed file system usually adopts a synchronous mode, the application needs to wait for the response of the metadata server before writing data, thus affecting the efficiency of the entire write operation

In the application scenario of data acquisition, the amount of data that needs to be written per second is very large, and this effect can easily cause the data generation speed to be greater than the writing speed, resulting in the inevitable loss of some data

Images