A load balancing-based query-type deduplication method and device
A technology for deduplication and load balancing, applied in the communication field, which can solve the problems of a large amount of query time, poor scalability of stateless routing policies, and occupation of stateful routing policies.
Inactive Publication Date: 2021-05-11
NORTHWESTERN POLYTECHNICAL UNIV
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] Embodiments of the present invention provide a query-type deduplication method and device based on load balancing, which are used to solve the problem of poor scalability of stateless routing strategies in the prior art, stateful routing strategies occupy a large amount of memory, and require a large amount of query time The problem
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0036]The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0037] figure 1 A schematic flow chart of a non-query deduplication method provided by an embodiment of the present invention, as shown in figure 1 As shown, the method mainly includes the following steps:
[0038] Step 101, from the data blocks obtained by dividing the data stream into blocks, super blocks and the fingerprints corresponding to each of the data blocks, determine a plurality of the data blocks with the smallest fingerprints, and determine accor...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a method and device for deduplication of query-type data based on load balancing, and relates to the technical field of communication. The method includes: determining a plurality of storage nodes corresponding to a plurality of minimum fingerprints from the data blocks obtained by dividing the data stream into blocks, super blocks and fingerprints corresponding to each of the data blocks; The number of matches with multiple storage nodes, when it is determined that the number of matches with multiple storage nodes is non-zero, determine the first storage node according to the determined number of matches with multiple storage nodes and the capacity of the storage nodes; send the super block To the first storage node, select the container number corresponding to the minimum fingerprint from the first storage node according to the data block with the minimum fingerprint selected from the super block; when it is determined that the minimum fingerprint corresponding to the container number and the minimum fingerprint corresponding to The data block is deleted when it is stored in the cache database.
Description
technical field [0001] The present invention relates to the technical field of communication, and more specifically, to a method and device for deduplication of query-type data based on load balancing. Background technique [0002] With the popularization of information technology and the continuous development of the Internet, society is entering an era of rapid data growth, and more and more data needs to be managed, and there are a lot of duplicate data in these data, so the storage of data causes a lot of storage waste . Data deduplication technology is a special data compression method, and data deduplication technology compresses data in units of files or data blocks. A single node is no longer able to handle a large amount of data. Currently, the cluster data deduplication technology is widely used. There are a large number of data storage nodes in the cluster. Therefore, how to reasonably distribute the uploaded data to these storage nodes is very important for the ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06
CPCG06F3/0607G06F3/0641G06F3/067
Inventor 蒋泽军王丽芳杜承烈刘志强范刚龙褚伟波尤涛陈进朝史豪斌潘炜赵正伟邓磊罗立志
Owner NORTHWESTERN POLYTECHNICAL UNIV