Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Distributed storage method and device

A distributed storage and storage table technology, applied in the storage field, can solve the problems of large space occupied by data index, small storage space, and inability to realize parallel reading of multiple storage nodes, so as to achieve small space occupied by data index and short data path Effect

Inactive Publication Date: 2019-05-24
ZHEJIANG UNIVIEW TECH CO LTD
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The disadvantage of the existing solution is that a single data file is used as the storage granularity to write to the file system, and the storage granularity is too large. If the storage space occupied by the data segment to be read is small, the data segment may all be stored in the file of a storage node on the system
When reading data segments, parallel reading of multiple storage nodes cannot be realized, and the data reading speed is limited by the capacity of the storage node, and the reading performance is not good
In addition, the file system is used to store data, and the data index occupies a large space

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed storage method and device
  • Distributed storage method and device
  • Distributed storage method and device

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0051] Please refer to image 3 and Figure 4 , image 3 is one of the flow charts of the steps of the distributed storage method provided in the first embodiment of the present invention, Figure 4 It is a schematic diagram of the data distributed storage solution provided by the first embodiment of the present invention. Wherein, each storage node 400 includes at least one resource block 410, and the number of resource blocks 410 attached to each storage node 400 can be set according to actual needs.

[0052] The flow of the distributed storage method will be described in detail below. The method includes: step S110, step S120, step S140 and step S150.

[0053] Step S110, dividing the original data to obtain at least one data file.

[0054] In this embodiment, the raw data is first obtained from the collector 140, and the raw data is divided into at least one such as Figure 4 The data files shown are fixed-size files. Wherein, the file size of each data file can be d...

no. 2 example

[0080] Please refer to Figure 10 , Figure 10 It is a flowchart of the steps of the distributed storage method provided by the second embodiment of the present invention. The distributed storage method is applied to a monitoring system 10 , and the monitoring system 10 includes: a management server 300 , at least one data collection device 100 , multiple storage nodes 400 and at least one terminal device 500 connected in communication. Each storage node 400 includes at least one resource block 410 . The method includes: step S210, step S220, step S230, step S240, step S250, step S260 and step S270.

[0081] In step S210, the management server 300 or the data collection device 100 divides the original data to obtain at least one data file.

[0082] Step S220, the management server 300 or the data collection device 100 cuts the data file according to the data content of the data file to obtain multiple granularity blocks.

[0083] Step S230, the management server 300 or the...

no. 3 example

[0090] Please refer to Figure 11 , Figure 11 A block diagram of function modules of the distributed storage device 200 provided for the third embodiment of the invention. The distributed storage device 200 is applied to a management server 300 or a data collection device 100 that is communicatively connected to a plurality of storage nodes 400 . The devices include:

[0091] A division module 210, configured to divide the original data to obtain at least one data file;

[0092] The cutting module 220 is configured to cut the data file according to the data content of the data file to obtain multiple granularity blocks.

[0093] The acquiring module 240 is configured to acquire the storage address corresponding to each granularity block.

[0094] The index generation module 250 is configured to generate a corresponding index record according to the file parameters of the data file and the acquired storage address corresponding to each granularity block, and write the inde...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a distributed storage method and device. The method comprises the following steps: dividing original data to obtain at least one data file; and cutting the data file according to the data content of the data file to obtain a plurality of granularity blocks; and obtaining a storage address corresponding to each granularity block. And generating a corresponding index record according to the file parameters of the data file and the acquired storage address corresponding to each granularity block, and writing the index record into an index storage table. Therefore, by further cutting the data file, one data file is discretely stored in a plurality of storage nodes, so that multi-node parallel reading can be realized. Moreover, a block system is adopted to store data, direct reading and direct storage can be achieved through an ISCSI protocol. The data transmission path is the shortest, and the occupied space of the data index is small.

Description

technical field [0001] The present invention relates to the technical field of storage, in particular to a distributed storage method and device. Background technique [0002] In the prior art, multiple file systems, such as FS1 and FS2, exist on the storage node. The original data can be divided into multiple data files with a certain file size (for example, 1GB), and the multiple data files can be discretely stored in the file systems of multiple storage nodes according to a certain strategy. For example, multiple data files may be stored cyclically in the order of FS1 of storage node 1, FS1 of storage node 2, and FS2 of storage node 3. [0003] The disadvantage of the existing solution is that a single data file is used as the storage granularity to write to the file system, and the storage granularity is too large. If the storage space occupied by the data segment to be read is small, the data segment may all be stored in the file of a storage node on the system. When...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F3/06
Inventor 吕志明
Owner ZHEJIANG UNIVIEW TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com