File processing method and device based on erasure codes and computer equipment

A file processing and erasure code technology, which is applied in computing, file system, file access structure, etc., can solve problems such as unable to process file encoding

Active Publication Date: 2020-11-13
PINGAN INT SMART CITY TECH CO LTD
View PDF6 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The embodiment of the present invention provides a file processing method, device, and computer equipment based on erasure codes, which solves the problem that in the existing small and medium file storage system FastDFS, files in the FastDFS system cannot be encoded by erasure code technology to The problem of making full use of disk storage

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • File processing method and device based on erasure codes and computer equipment
  • File processing method and device based on erasure codes and computer equipment
  • File processing method and device based on erasure codes and computer equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0032] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude one or Presence or addition of multiple other features, integers, steps, operations, elements, components and / or collections thereof.

[0033] It should also be understood that the terminology used ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a file processing method and device based on erasure codes and computer equipment, and the method is applied to a FastDFS system, and comprises the steps: obtaining a cold filein a hot data set of a storage server; partitioning the cold file according to a preset partitioning rule to obtain file blocks corresponding to the cold file; encoding the file block according to anerasure code technology to obtain a check block corresponding to the file block; storing the file block and the check block into a cold data set of the storage server; and if a downloading request for downloading the cold file is received, obtaining the cold file from the cold data set according to the downloading request. Based on the hierarchical storage technology, the files stored in the FastDFS system are optimized to improve the storage utilization rate of the disk, the storage cost is reduced, and the operation and maintenance efficiency is improved.

Description

technical field [0001] The invention belongs to the technical field of distributed storage, and in particular relates to an erasure code-based file processing method, device and computer equipment. Background technique [0002] FastDFS is an open source lightweight distributed file storage system developed in C language, which consists of Tracker Server, Storage Server and Client. Among them, the storage server is divided into several groups for actually storing data, and the storage in the group backs up each other to ensure high availability of data. Since the N storages in the same group have the same data, it can tolerate N-1 storage failures, but its storage utilization rate is 1 / N. Distributed file storage systems such as Hdfs and Ceph use erasure code technology to encode n original data blocks to generate m check blocks, and store these n+m data blocks in the system. While the block is wrong, the system is guaranteed to be available, which reduces the storage effic...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F11/10G06F16/13G06F16/16G06F16/182
CPCG06F11/1004G06F16/134G06F16/162G06F16/182
Inventor 赵芳明
Owner PINGAN INT SMART CITY TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap