Method for acquiring self-increment mass data files from multiple sources

Abstract

The invention relates to a method for acquiring self-increment mass data files from multiple sources. In the method, the multithreading technology is utilized for acquiring self-increment data files in parallel from data sources, and the file slice technology and file continuous transferring technology are utilized for slicing the self-increment data files according to time, the increment of the self-increment data files is acquired each time. The acquiring time intervals are set according to a data generating circle, data file size estimation and business requirements, and current cyclic data files from server data source are checked by polling at regular intervals according to the acquiring time intervals, increment data are collected by aid of the file slice technology and the file continuous transferring technology to store locally in a form of small data files and to record current-time file byte size as an initial position of next polling acquirement. By acquiring the increment each time, the method realizes self-increment mass data files acquirement, so that the technical problems of long delay time, poor real-time performance, server load and stability influencing elements in collecting teledata of the current technology are solved.

Description

Technical field: [0001] The invention relates to the field of physics, in particular to a mass data collection technology in a computer application system, in particular to a method for real-time collection of multi-source multi-source massive data files. Background technique: [0002] The amount of data involved in telecommunication services is huge. In a large telecom application system, there are usually multiple data sources simultaneously providing massive real-time self-increasing data files, and the application system needs to collect tens to hundreds of GB of massive data every day, such as PCMD and ROP data. This type of data is stored in multiple server data sources in the form of files. Each data source generally generates a file at a fixed time period, for example, one data file is generated every hour or one data file is generated every day. Increment until the start of the next cycle, the corresponding data file will be automatically created and self-increment...

AI Technical Summary

Problems solved by technology

[0004] The object of the present invention is to provide a kind of real-time collection method of multi-source increasing massive data file, described this multi-source real-time collecting method of increasing massive data file will solve the long delay time of collecting telecommunication data in the prior art, real-time performance is poor, influence Technical issues with server load and stability

Images