Load balance-based query type repeated data deletion method and device
A technology for deduplication and load balancing, applied in the field of communications, which can solve problems such as large memory, occupation of stateful routing policies, and poor scalability of stateless routing policies.
Inactive Publication Date: 2018-05-29
NORTHWESTERN POLYTECHNICAL UNIV
View PDF6 Cites 10 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] Embodiments of the present invention provide a query-type deduplication method and device based on load balancing, which are used to solve the problem of poor scalability of stateless routing strategies in the prior art, stateful routing strategies occupy a large amount of memory, and require a large amount of query time The problem
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0036]The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0037] figure 1 A schematic flow chart of a non-query deduplication method provided by an embodiment of the present invention, as shown in figure 1 As shown, the method mainly includes the following steps:
[0038] Step 101, from the data blocks obtained by dividing the data stream into blocks, super blocks and the fingerprints corresponding to each of the data blocks, determine a plurality of the data blocks with the smallest fingerprints, and determine accor...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a load balance-based query type repeated data deletion method and device, and relates to the technical field of communication. The method comprises the following steps of: determining a plurality of storage nodes corresponding to a plurality of minimum fingerprints from data blocks obtained through blocking a data flow, a super block and fingerprints corresponding to the data blocks; determining a quantity matched with the plurality of storage nodes through a Bloom filter, and when the quantity matched with the plurality of storage nodes is not zero, determining a firststorage node according to the determined quantity matched with the plurality of storage nodes and capacities of the storage nodes; sending the superblock to the first storage node, and selecting container numbers corresponding to the minimum fingerprints from the first storage node according to the data blocks, with the minimum fingerprints, selected from the super block; and when determining that the minimum fingerprints corresponding to the container numbers and the data blocks corresponding to the minimum fingerprints are stored in a cache database, deleting the data blocks.
Description
technical field [0001] The present invention relates to the technical field of communication, and more specifically, to a method and device for deduplication of query-type data based on load balancing. Background technique [0002] With the popularization of information technology and the continuous development of the Internet, society is entering an era of rapid data growth, and more and more data needs to be managed, and there are a lot of duplicate data in these data, so the storage of data causes a lot of storage waste . Data deduplication technology is a special data compression method, and data deduplication technology compresses data in units of files or data blocks. A single node is no longer able to handle a large amount of data. Currently, the cluster data deduplication technology is widely used. There are a large number of data storage nodes in the cluster. Therefore, how to reasonably distribute the uploaded data to these storage nodes is very important for the ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F3/06
CPCG06F3/0607G06F3/0641G06F3/067
Inventor 蒋泽军王丽芳杜承烈刘志强范刚龙褚伟波尤涛陈进朝史豪斌潘炜赵正伟邓磊罗立志
Owner NORTHWESTERN POLYTECHNICAL UNIV