Data storage and recovery method, codec and codec system
A data storage and codec technology, applied in electrical digital data processing, redundancy in computing, data error detection, instrumentation, etc. It can solve problems such as peak network resource occupation, overall system performance impact, and CPU resource consumption. , to achieve the effect of improving the reading speed, reducing resource consumption, and reducing the occupation of network resources
Pending Publication Date: 2020-12-08
JIANGSU SUPERFLUIDITY INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] In the process of implementing the present invention, the inventors found that the EC mechanism uses matrix multiplication operations in the process of encoding and decoding, so although EC codes can greatly save space, additional calculations will be generated in the process of storage and restoration Especially for systems with a large amount of data to be processed, when a storage node fails and triggers data recovery, a large number of files will be decoded. At this time, a large amount of CPU resource consumption will often be generated, which may affect the overall system Performance hits, slowing down normal computing tasks
Especially during data recovery, computing nodes need to read data blocks and check blocks of other nodes, which may cause a peak in network resource usage in a short period of time
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0026] The specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific embodiments.
[0027] Unless expressly stated otherwise, throughout the specification and claims, the term "comprise" or variations thereof such as "includes" or "includes" and the like will be understood to include the stated elements or constituents, and not Other elements or other components are not excluded.
[0028] In order to overcome the problems in the prior art, the present invention firstly provides a data storage method, such as Figure 4 As shown, the data storage method in one embodiment includes: step S10 to step S14.
[0029] File cutting in step S10: cutting the original file to be stored into multiple file segments. Preferably, cutting the original file to be stored so as to cut it into multiple file segments incl...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a data storage and recovery method, a codec and a codec system. The data storage method comprises the steps of cutting an original file to be stored into a plurality of file segments; determining a generation matrix according to the number of preset data blocks and the number of preset check blocks; dividing the generation matrix into a plurality of low-dimensional sub-matrixes; based on an erasure code encoding algorithm, performing batch encoding on each file segment by adopting the low-dimensional sub-matrix; and carrying out distributed storage on each coded file segment. According to the data storage and recovery method, the codec and the codec system, the data processing speed can be improved.
Description
technical field [0001] The invention relates to the field of data storage, in particular to a data storage and restoration method, codec and codec system. Background technique [0002] In recent years, Erasure Coding (EC), as an emerging data storage scheme, has gradually become one of the mainstream schemes for large-scale data storage. Through the EC mechanism, the data is divided into pieces and stored in different node hard disks according to a certain redundancy ratio, which ensures that data will not be lost when some devices fail, and also saves storage space, which is better. It solves the problems of data storage reliability and space utilization effectively. The following is a brief description of the mainstream RS (Reed-Solomon) type erasure code mechanism in EC codes. [0003] There are two core parameters m and n in this type of erasure code, which respectively represent the number of check blocks and the number of original data blocks. The erasure code encod...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/10G06F11/14
CPCG06F11/1004G06F11/1469
Inventor 吕川张晓星张炜
Owner JIANGSU SUPERFLUIDITY INFORMATION TECH CO LTD