Working method of high-reliability distributed log system

Abstract

The invention provides a working method of a high-reliability distributed log system. The method comprises the steps of providing a distributed file system, wherein the distributed file system has several storage nodes, multiple writers and multiple readers; adopting the writers to add records to a log file simultaneously to generate a write-in request of the log file; receiving the writ-in request of the log file, and allocating a serial number to the log file; generating multiple copies for the log file allocated with the serial number, randomly writing each copy into a storage node of the distributed file system, and transmitting the copies to all the readers corresponding to the log through the distributed file system. By means of the working method of the high-reliability distributedlog system, coexistence of the availability and consistency can be achieved, high availability is achieved, that is, the service is always on line, the off line state is not allowed to appear, the write-in response is quick, a certain degree of data reading exists, and generally, recently write-in log data is read; meanwhile, the working method has durability, and the data loss probability is lowered.

Description

technical field [0001] The invention relates to the field of distributed systems, and belongs to a working method of a highly reliable distributed log system. Background technique [0002] With the deepening of the application of Internet technology, more and more log data are generated by various application systems, and the application of log data is becoming more and more important. All information system platforms generate a large number of logs every day, usually streaming data, including user access records, database operation records, etc. When the amount of data reaches a certain order of magnitude, traditional single-node systems cannot complete retrieval and analysis tasks. They must be processed using a distributed logging system. At present, the mainstream log systems are all distributed systems. For distributed log systems, it is extremely difficult to meet the three requirements of strong consistency, availability, and partition fault tolerance while processin...

AI Technical Summary

Problems solved by technology

At present, the mainstream log systems are all distributed systems. For distributed log systems, it is extremely difficult to meet the three requirements of strong consistency, availability, and partition fault tolerance while processing high-concurrency transactions under large-scale clusters. At present, Most distributed log systems can only satisfy two of them at most, or prefer two of them

The main difficulties are: (1) In a distributed system, satisfying strong consistency will inevitably increase the response delay of the client, which will reduce the availability

(2) In a distributed system, if you want to have higher availability, you must not satisfy strong consistency, but only final consistency

(3) If you want to meet strong consistency and availability at the same time, the best way is to use a single node, but this will cause a single point problem, and the partition fault tolerance cannot be satisfied

Another example is the ELK (distributed search engine, collection tool, display tool) log system, which cannot be persistent in terms of data storage. The distributed search engine cannot really store data as a file system, and there is a risk of data loss.

Images