A distributed system data collection method and data signature generation method

Abstract

The invention provides a distributed system data collection method. The method is characterized in that updated file contents are divided into more than 2 parts according to a flow control logic and configurations, an encryption function is called for each part, and data is encrypted and then reported, wherein the flow control logic is as follows: all the data to be reported is segmented accordingto the maximum transmission unit (MTU) configured by a user, a signature function is then called for each segment, and signatures are generated for the data and added to the end of the data. Comparedwith the prior art, the distributed system data collection method provided by the invention has the advantages that distributed data collection can be guaranteed without excessive pressure on a system bandwidth, a log system can be preserved to the largest extent, and a certain degree of information security guarantee can be further provided.

Description

technical field [0001] The invention relates to a method for collecting data in a distributed system and a method for generating a data signature, in particular to a method for collecting data in a distributed system and a method for generating a data signature suitable for collecting a large amount of data. Background technique [0002] With the expansion of the scale of Internet enterprises, the amount of data and data sources they face will increase exponentially, and the complexity of the system will also rise sharply. The corresponding demand for safe data collection will also greatly increase. [0003] Today's mainstream monitoring data acquisition framework more or less draws on the design concept of the Log4j library. Log4j implements a Java-based data logging tool framework, which was originally designed and developed by Ceki, and has now become an open source project of Apache. Three important concepts are defined in Log4j, namely Logger, Appender and Layout: Logg...

AI Technical Summary

Problems solved by technology

[0006] It can be seen that although various existing data acquisition modules can realize the output of monitoring information, they are relatively simple in design.

It is impossible to collect distributed information together, nor can it guarantee the safety and reliability of information, nor can it control the flow according to the overall status of the system. Therefore, in the real-time operation of the application system, if a large amount of data write requests suddenly appear, it will inevitably lead to performance bottlenecks , ranging from data loss to system failure

Moreover, the distributed system has a complexity that cannot be compared with a single computer, and network bandwidth resources are even more important. How to help the monitoring system accurately find the fault site among a large number of servers, these traditional data collection methods are also incompetent

[0007] To sum up, although there are many monitoring data asynchronous collection systems, they cannot provide a safe and stable solution based on the overall bandwidth and performance of the distributed system, and the defects are obvious.