Data distributed storage system and method

A distributed storage and data grouping technology, applied in the field of database storage, can solve problems that are not suitable for data access and processing, and achieve the effect of improving response time and ensuring security

Inactive Publication Date: 2014-10-29
SHANGHAI PEOPLEYUN INFORMATION TECH CO LTD
View PDF4 Cites 39 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Using the database storage method of the distributed file system, although it solves the data security and the access requirements of massive data, it is not suitable for those data access and processing that require low latency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data distributed storage system and method
  • Data distributed storage system and method
  • Data distributed storage system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0048] see figure 1 , the present invention discloses a data distributed storage system, which includes: a registration module 1 (also called a "node cluster module"), a data import module 2, a storage module 3, a backup module, and a retrieval module 4.

[0049] The registration module 1 is used to register the data nodes in the cluster to the management node through the client;

[0050] The data import module 2 is used to scan the input data according to the set size of the data block and load it into the memory, the data in the memory is grouped according to the characteristic value of the data, and then the grouped data is sent to the corresponding data node.

[0051] Specifically, see image 3 , in this embodiment, the data import module specifically includes a data segmentation unit, a file scanning unit, a data grouping rule matching unit, a data grouping unit, and a data sending unit.

[0052] The data segmentation unit is used to scan the input data according to the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data distributed storage system and method. The system comprises a node cluster module, a data import module and a storage module. The node cluster module is used for connecting data nodes in a cluster with corresponding management nodes; the data import module is used for scanning input data according to data blocks with sizes set and loading the input data into a memory, data in the memory are grouped according to characteristic values of the data, and the grouped data are sent to the corresponding data nodes; the storage module is used for storing data fragmentations in the memory after the data nodes receive file fragmentations, and the data nodes output logs to a hard disk; whether the data in the memory exceed a set threshold value or not is judged, if the data in the memory exceed the set threshold value, the data are reorganized and compressed and then written into the hard disk, and corresponding log files restored through user memory data are deleted. According to the system and method, the cluster based on memory computing power can be accelerated; real-time loading and processing capacity for large-scale data can be improved, and response time of the system is shortened.

Description

technical field [0001] The invention belongs to the technical field of database storage, and relates to a distributed storage system, in particular to a data distributed storage system; meanwhile, the invention also relates to a data distributed storage method. Background technique [0002] At present, the data storage methods of the database include: 1. Stand-alone data storage method; 2. Master-slave backup storage method; 3. Storage method using distributed file system. However, no matter which of the above methods is adopted, there are certain deficiencies. [0003] Although the stand-alone data storage method is easy to manage and use, it has serious defects in scalability and cannot meet the access needs of today's massive data, and there are also problems in data security. The master-slave backup storage method only solves the security problem, and other problems still exist. Although the database storage method using the distributed file system solves the data secu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/182
Inventor 吴朱华潘志铭
Owner SHANGHAI PEOPLEYUN INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap