Data processing method, system and apparatus

A data management and data block technology, applied in the field of computing, can solve the problems of large data storage redundancy, low efficiency, and inability to play EC coding, etc., and achieve the effect of simplifying addressing operations and ensuring reliability

Active Publication Date: 2018-04-06
HUAWEI TECH CO LTD
View PDF5 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this solution is inefficient when dealing with small data objects, because in small data objects, the data lengths of Metadata, Key and Value are not much different, and even the data lengths of Metadata and Key are greater than the data length of Value
According to the information released by Facebook, most of the data objects stored in memory storage are small data objects, and even more than 40% are smaller than 11bit, indicating that most of the data are small data, and this kind of partial code storage The scheme cannot take advantage of EC coding, which makes the data storage redundant and increases the storage cost.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, system and apparatus
  • Data processing method, system and apparatus
  • Data processing method, system and apparatus

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0068] The following describes the composition of the distributed storage system. figure 1 is a schematic architecture diagram of the distributed storage system 100 . refer to figure 1 , the distributed storage system 100 includes the following key entities: client 110, storage node 130, coordination manager 150, and backup node 170, these key entities are connected through a network, for example, through a long-term transmission control protocol (Tr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method. The method comprises the steps of coding a data block with a predetermined size to generate an error correction data block corresponding to the data block, wherein the data block comprises data objects, and the data objects include a keyword, a value and metadata; and generating a data block index and data object indexes, wherein the data block index is used for indexing the data block and the error correction data block corresponding to the data block, and the data object indexes are used for indexing the data objects in the data block, and each data object index is used for indexing only one data object.

Description

technical field [0001] The present invention relates to the computing field, and more particularly, to a data processing method, system and device. Background technique [0002] With the decline of memory prices, distributed memory storage systems are widely used in distributed computing systems to store hot data, among which key-value (KV) pair storage is the most widely used data storage method, currently Mainstream commercial products include Memcached, Redis, RAMCloud, etc., which are commercially used in the data storage systems of Twitter, Facebook, and Amazon. [0003] The mainstream fault-tolerant method of the distributed memory storage system is mainly the full backup scheme, which is to copy a piece of data to different devices as a whole. When some devices fail, the backup data on other non-failed devices can be used to restore Recover data from failed devices. This implementation scheme is simple and reliable, but there is a large data redundancy, and at least...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/10
CPCG06F11/1016G06F11/1044G06F11/1076G06F12/0246G06F2212/7207G06F3/0673G06F3/0644G06F3/0619G06F12/02
Inventor 张家劲姚文东李柏晴
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap