Storage method of mass filing stream data

A streaming data, mass technology, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problem of difficult to meet the data recovery needs of large-scale systems, the large scale of archived streaming data storage systems, and the large scale of data to be recovered, etc. problems, to achieve the effect of ensuring data loading performance, ensuring data reliability, and reducing the scale of the system

Inactive Publication Date: 2012-07-04
NAT UNIV OF DEFENSE TECH
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the scale of the archive stream data storage system is large, and there are problems of high failure rate and large scale of data to be recovered. The traditional logical serial data recovery method is difficult to meet the data recovery needs of large-scale systems, so a new data recovery method is needed.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Storage method of mass filing stream data
  • Storage method of mass filing stream data
  • Storage method of mass filing stream data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0045] specific implementation plan

[0046] figure 1 It is an overall flow chart of the present invention, which is specifically divided into the following steps:

[0047] 1. Build a loading pool and an archive pool.

[0048] 2. Generate MiniRDB periodically and store MiniRDB in the archive pool.

[0049] 3. Use the parallel distribution recovery method to restore MiniRDB.

[0050] figure 2 It is a structural diagram of the database cluster in the loading pool designed in the first step of the present invention; the loading pool used for real-time storage of archive stream data is composed of multiple database clusters connected in series. Each database cluster consists of a group of mutual backup database nodes and a coordinating node connected through Ethernet. The coordination node is a computer that distributes the user's loading request to all database nodes according to the load balancing strategy, and forwards the user's query request to a certain database node. ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a storage method of mass filling stream data, aiming at solving the storage of mass filing stream data and improving the management scale, access performance and reliability of the filling stream data according to the characteristics of filling stream data. The technical scheme is as follows: firstly building a loading pool which stores the filing stream data generated at high speed in real time and a filing pool which stores larger-scale filing data transferred from the loading pool; periodically generating read-only database file MiniRDB and storing MiniRDB into the filing pool; and when nodes of a plurality of filing databases in the filing pool are in fault, adopting a parallel distributed recovery method for recovering. Compared with the prior art, by adoptingthe invention, the scale of data capable of being stored and managed is larger, the storage and enquiry performances of data are higher, and the reliability of data is higher.

Description

technical field [0001] The invention relates to a data storage method in the computer field, in particular to a method for storing massive archiving flow data based on a computer cluster system. Background technique [0002] With the development of computer network and storage technology, a new demand for mass data storage and analysis has emerged in many fields such as network security management, network search, communication, finance, and scientific experiments. Data, and mining the regular knowledge hidden in this type of data. For example, in network security applications, real-time storage, query and analysis of network communication packet header information is a common network security management method. Generation, the generation speed is fast (the number of messages generated by an ordinary Gbit network connection can reach 100,000 per second, and its data generation speed can reach dozens of MB per second), so the formed system is large in scale. Similar applica...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30H04L29/08
Inventor 杜凯王怀民杨树强
Owner NAT UNIV OF DEFENSE TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap