Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data truncation method and device based on erasure codes

A technology for data truncation and erasure coding, applied in the field of data truncation based on erasure coding, can solve the problems of truncation operation amplification, affecting the performance of erasure coding, reducing the write performance of distributed storage systems, etc. The effect of improving write performance

Active Publication Date: 2018-09-28
NEW H3C TECH CO LTD
View PDF17 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, due to the need to consider the impact of other write transactions on the truncation operation, it is necessary to do additional fault-tolerant processing on the truncation operation, resulting in the amplification of the truncation operation
The amplification of the truncation operation includes the backup and deletion of some data in the data to be truncated, which affects the performance of the erasure code and reduces the write performance of the distributed storage system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data truncation method and device based on erasure codes
  • Data truncation method and device based on erasure codes
  • Data truncation method and device based on erasure codes

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0026]The distributed storage system stores the data that needs to be stored sent by the user through the client in a distributed manner on multiple storage nodes. In order to enhance the reliability of the distributed storage system and the ability to To migrate data from one storage node to another, in order to facilitate data migration between storage nodes, users need to organize, migrate and maintain the stored data in units of data maintenance units, including data maintenance after abnormal recovery of storage nodes Data recovery is also performed in units of data maintenance units. The data that users need to store will first be divided into several data maintenance units, and each data maintenance unit will be stored in different storage nodes. Therefore, the storage node is an indispensable component in the distributed storage system, and the data maintenance unit is also an indispensable concept in the distributed storage system. However, for different specific dis...

Embodiment approach

[0037] As an implementation manner, the method for executing the data truncation command may include:

[0038] First, the OSD node 10 generates a log version number according to preset rules. The log version number is the unique number of the operation log of the data truncation command, and each data truncation command corresponds to a log version number.

[0039] Secondly, the OSD node 10 obtains the slave OSD node list that stores the data to be truncated corresponding to the object according to the object ID, that is, the OSD node 30, OSD node 40, etc. slave OSD node lists, and the log version number, the object in the data truncation command The ID and the truncation position in the object are sent to the OSD node 30, OSD node 40 and other slave OSD nodes, so that the OSD node 30, OSD node 40 and other slave OSD nodes execute the data truncation command. At the same time, the OSD node 10 executes the data truncation command to truncate the data to be truncated correspond...

no. 2 example

[0056] Please refer to Figure 6 , Figure 6 A schematic block diagram of an erasure code-based data truncation device 200 provided by an embodiment of the present invention is shown. The data truncation device 200 based on an erasure code is applied to a storage node in a distributed storage system, and includes a receiving module 201 , a judgment module 202 , a data truncation module 203 , a determination module 204 and a control module 205 .

[0057] The receiving module 201 is configured to receive a data truncation command sent by the client, wherein the data truncation command is generated by the client in response to the file truncation request to truncate the data to be truncated stored in the storage node.

[0058] In the embodiment of the present invention, the receiving module 201 may be used to execute step S101.

[0059] The judging module 202 is configured to judge whether there is a data writing transaction being processed on the data maintenance unit to which...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the field of distributed storage technology, and provides a data truncation method and device based on erasure codes. The method includes: receiving a data truncation commandsent by a client, wherein the data truncation command is generated by the client in response to a file truncation request to truncate to-be-truncated data stored in a storage node; judging whether a currently processed data writing transaction exists on a data maintenance unit to which the to-be-truncated data belong, and if yes, waiting for completion of processing on the currently processed datawriting transaction, and executing the data truncation command thereafter; and when no currently processed data writing transaction exists on the data maintenance unit to which the to-be-truncated data belong, executing the data truncation command. According to the method, it is guaranteed that no currently executed data writing transaction exists before the truncation command is executed, thus backing-up and deletion operations on partial data in the to-be-truncated data are avoided, the writing amplification problem in data truncation is solved, and writing performance of a distributed storage system is improved.

Description

technical field [0001] The present invention relates to the technical field of distributed storage, in particular to a data truncation method and device based on erasure codes. Background technique [0002] In distributed storage systems, erasure codes have been applied to object storage, block storage, and file system storage. Erasure codes need to perform distributed fault-tolerant processing on all write operations (including truncation operations) during business processing to prevent node failures from destroying data. However, since the impact of other write transactions on the truncation operation needs to be considered, additional fault-tolerant processing must be performed on the truncation operation, resulting in the amplification of the truncation operation. The amplification of the truncation operation includes backup and deletion of some data in the data to be truncated, thus affecting the performance of the erasure code and reducing the writing performance of ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F11/07G06F11/10
CPCG06F11/0727G06F11/1004
Inventor 王文锋
Owner NEW H3C TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com