Unlock instant, AI-driven research and patent intelligence for your innovation.

A fog storage data redundancy elimination method

A data storage and de-redundancy technology, applied in database indexing, database updating, other database retrieval, etc., can solve problems such as node data redundancy, reduce I/O and improve query speed.

Pending Publication Date: 2021-07-02
国网新疆电力有限公司信息通信公司 +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The present invention provides a method for de-redundancy of fog storage data, which overcomes the shortcomings of the above-mentioned prior art, and can effectively solve the problem of node data redundancy

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A fog storage data redundancy elimination method
  • A fog storage data redundancy elimination method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0042] Embodiment 1: The fog storage data de-redundancy method includes the following steps:

[0043] Step S1, the mobile terminal makes a query (storage) request to the administrator of the fog node;

[0044] Step S2, the fog administrator checks the statistical content of the fog process log management server, which are the virtual machine resource occupation table and the service (request) process table;

[0045] Step S3, when the available capacity of the fog node satisfies the request of the mobile terminal, the fog computing (storage) server will process the request, de-duplicate and store (query) the data flow incoming to the fog node;

[0046] Step S4, when the available capacity of the fog node satisfies the request of the mobile terminal, the fog computing (storage) server will process the request, and at the same time, the fog administrator will update the current task processing process, and the fog process log management server will update the resource occupation ...

Embodiment 2

[0047] Embodiment 2: as the optimization of embodiment 1, such as figure 1 and figure 2 As shown, the specific implementation of step S3 is:

[0048] S31, incoming data stream chunking

[0049] The data transmitted from the mobile terminal to the fog node is transmitted to the fog (cloud) computing (storage) server, and the data flow is divided into equal length blocks;

[0050] S32, calculate the data block fingerprint

[0051] Select the MD5 algorithm that generates a smaller fingerprint, and perform hash calculation on the divided data blocks to obtain the fingerprint of each data block. Stored in memory as metadata for each data block;

[0052] S33, calculate the index value

[0053] Calculate the hash of the obtained fingerprint again, get the index (integer), and map it to a fixed-size index table. Each index value corresponds to multiple fingerprints. The red-black tree (RBTree) is used as the data storage structure of the fingerprint, and other in-memory Compare...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of redundant data processing methods, in particular to a fog storage data redundancy elimination method, which is used for processing a query or storage request of a mobile terminal according to the available capacity of a fog node. The fog storage data redundancy elimination method is a deletion scheme for low-delay duplicated data, and includes building an index table in a memory, in which a red-black tree corresponding to each index value serves as a structure for storing data fingerprints, so that I / O between a disk and the memory is reduced, and the query speed is increased; Judging whether the data blocks with the same data fingerprints are repeated or not by using a cyclic redundancy code (CRC) technology, and storing conflict data blocks in fingerprint nodes by using a linked list structure. Besides, in order to prevent sudden crash of the operating system, the fingerprint table is persistently stored in the memory; once the operating system crashes and is restarted, data in the memory disappears, at the moment, the two files in the disk are combined to reconstruct the fingerprint table in the memory, and contents of the two files are emptied and recorded again at the same time.

Description

technical field [0001] The invention relates to the technical field of redundant data processing methods, and relates to a method for removing redundancy of fog storage data. Background technique [0002] The explosive growth of digital information in today's society, the rapid increase and flood of data volume has posed a huge challenge to data storage management technology, how to efficiently manage and store data has become a research hotspot. In the primary storage system and secondary storage system produced by Microsoft and EMC, there are 50% and 85% redundant data respectively. As time goes by, the proportion of redundant data will increase exponentially. Enterprises are storing these data The overhead required is also huge. [0003] Such a huge amount of data poses challenges to traditional storage systems. So technicians turned to cloud storage systems with larger capacity and lower cost. However, cloud storage adopts a data outsourcing model. In order to reduce ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/215G06F16/22G06F16/901G06F16/23G06F11/14G06F16/2455G06F16/2453
CPCG06F16/215G06F16/2255G06F16/2282G06F16/9024G06F16/2358G06F11/1471G06F11/1474G06F16/24568G06F16/24532G06F2201/80
Inventor 徐健黄强鲁学仲李浩升王庆鹏赵梅马怡璇陈伟
Owner 国网新疆电力有限公司信息通信公司