Method for time-sequence data processing, device and system therefor
A data processing device and data processing technology, applied in the direction of input/output to record carrier, etc., can solve the problems of reducing the data processing efficiency of the front-end application system, long time for basic file operations, long time for data processing, etc.
Active Publication Date: 2011-09-14
BEIJING 21VIANET DATA CENT
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
For the above-mentioned data management requirements, it is difficult to classify by hotspots. The main problem is that the access rate of data is generally low, and re-access occurs occasionally, and there is almost no situation where periodic partial data is repeatedly accessed.
[0005] The inventors found that: since all the data in the hierarchical storage is classified according to the access frequency, when the amount of data is large and the stored data does not have the characteristics of hotspots, an online storage system with a large capacity is required to support it, which is often It will make the basic operation of the file take a long time, which will cause the user to process the data for a long time, thereby reducing the data processing efficiency of the front-end application system
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0031] The basic idea of the embodiments of the present invention is to provide a sequential data processing method, device and system by fully considering the timing of processed data and combining data types during data processing, thereby improving data processing efficiency.
[0032] Specifically, considering the time sequence relationship and access characteristics of data production, adopting a time series-based hierarchical management method can effectively reduce the scale and cost of the online storage system. In the new storage system, a two-level storage system is adopted. The first-level storage system is used to store the latest generated data, and the second-level storage system is used to store data before a certain time threshold. systems, such as NAS, SAN, and even DSA systems, which use one or more cheap storage devices and form a logical closed-loop storage chain for management of the storage devices. When data is stored, it is first stored in the primary ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a method for time-sequence data processing, a device and a system therefor. The method comprises: acquiring time information, an attributive serialization passage and an application processing type of the data to be processed; corresponding the serialization passage to the application processing type; determining a storage unit corresponding to the data to be processed according to the time information and the serialization passage; and processing the data to be processed by the instruction of the application processing type by the storage unit. The method, the device and the system can optimize the storage and the management for the data.
Description
technical field [0001] The present invention relates to data storage and management technology, in particular to a sequential data processing method, device and system. Background technique [0002] With the continuous development of high-density mass data storage requirements, more and more applications use centralized storage systems to store and access various data, such as data backup, video surveillance, video editing, geological exploration, data archiving In the process of these data processing, they all have clear time characteristics, and have the following characteristics at the same time: [0003] The storage of the data is usually unstructured storage or near-unstructured storage, and the storage process requires relatively high IO bandwidth, such as tens of MB / s or more, and the total amount of data is relatively large and is generated continuously; and The access to the data usually has or implies time-series feature information; once the data enters the stora...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06
Inventor 杨耀敏蒋建平
Owner BEIJING 21VIANET DATA CENT