Unlock instant, AI-driven research and patent intelligence for your innovation.

A method, device and storage medium for data deduplication storage based on sliding window

A sliding window and data technology, applied in the direction of electronic digital data processing, digital data information retrieval, special data processing applications, etc., can solve the problem that single file storage cannot meet the performance requirements of large data volume storage, the overall cluster performance overhead increases, and time Problems such as thread growth in overhead, to achieve the effects of reducing time performance overhead, flexible query methods, and reducing storage pressure

Active Publication Date: 2020-12-01
SHENZHEN COOCAA NETWORK TECH CO LTD
View PDF10 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] With the development of science and technology and the increase of people's needs, the space required to store data is also increasing, but in practical applications, the storage space is limited
Every time a new file data is added, it is necessary to ensure the uniqueness of the data storage, and all files will be queried by establishing an index relationship. However, with the continuous addition of data, especially in the case of massive data, it takes a lot of At the same time, the traditional query method will cause the time overhead corresponding to the query to increase in threads, which will increase the performance overhead of the overall cluster, and the existing ordinary single-file storage can no longer meet the current requirements. The performance requirements of large data volume storage

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method, device and storage medium for data deduplication storage based on sliding window
  • A method, device and storage medium for data deduplication storage based on sliding window
  • A method, device and storage medium for data deduplication storage based on sliding window

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0052] See figure 1 , figure 1 It is a flow chart of the first preferred embodiment of the sliding window-based data deduplication storage method of the present invention.

[0053] Such as figure 1 As shown, a data deduplication storage method based on a sliding window includes the following steps:

[0054] In step S100, the stored data is obtained and recorded as a data source, and divided to obtain each piece of data and the optimal number of pieces of data.

[0055] In the present invention, the data source refers to data that presents the characteristics of streaming data, such as network data. According to the agreed channel, the qualified data is combined and transmitted in the channel, output one by one, and the data combination are sorted in the same order as they are exported. In the specific implementation, streaming storage data, such as massive file data, is obtained as a data source for deduplication query to optimize query and reduce overall performance overh...

Embodiment 2

[0089] The present invention also provides a data deduplication storage device based on a sliding window, such as Figure 4 As shown, the data deduplication storage device based on the sliding window includes a processor 10 and a memory 20 connected to the processor 10, the memory 20 stores a data deduplication storage program based on the sliding window, and the sliding window based The window data deduplication storage program is used by the processor 10 to implement the steps of the sliding window-based data deduplication storage method described in Embodiment 1, specifically as described above.

Embodiment 3

[0091] The present invention also provides a storage medium. The storage medium stores a data deduplication storage program based on a sliding window. When the data deduplication storage program based on a sliding window is executed by the processor 10, the data storage program based on a sliding window is realized Deduplication storage method; the details are as described above.

[0092] To sum up, the present invention discloses a data deduplication storage method, device and storage medium based on a sliding window. The data deduplication storage method based on a sliding window includes: obtaining stored data as a data source, and cutting points, to obtain each fragmented data and the optimal number of fragmented data; establish a query index for each fragmented data, and establish a variable sliding window; detect whether there is new data to be written, and if so, pass the The variable sliding window sends a query command to each segmented data after segmentation to dete...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data deduplication storage method and device based on a sliding window and a storage medium, and the method comprises the steps: carrying out the segmentation of stored data,and obtaining the number of each piece of fragmented data and an optimal piece of fragmented data; Establishing a query index for each piece of fragmented data, and establishing a variable sliding window; Detecting whether to-be-written new data exists or not, and if yes, sending a query instruction to each piece of segmented fragmented data through the variable sliding window to judge whether the to-be-written new data is repeated with the current piece of fragmented data or not, namely marking the to-be-written new data as repeated data; If yes, discarding the new data; And if not, writingthe new data into the current fragmented data. According to the method, the segmentation mode of the maximum number of fragments is determined from the time dimension and the spatial dimension according to the rule of occurrence of repeated data, and the fragment data is dynamically subjected to duplicate query and optimized query by adjusting the variable sliding window, so that the overall cluster performance cost is reduced, the query efficiency is improved, and convenience is brought to a user.

Description

technical field [0001] The present invention relates to the technical field of data storage, in particular to a sliding window-based data deduplication storage method, device and storage medium. Background technique [0002] With the development of science and technology and the increase of people's needs, the space required to store data is also increasing, but in actual use, the storage space is limited. Every time a new file data is added, it is necessary to ensure the uniqueness of the data storage, and all files will be queried by establishing an index relationship. However, with the continuous addition of data, especially in the case of massive data, it takes a lot of At the same time, the traditional query method will cause the time overhead corresponding to the query to increase in threads, which will increase the performance overhead of the overall cluster, and the existing ordinary single-file storage can no longer meet the current requirements. Performance requir...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/11G06F16/13G06F16/172
Inventor 赵磊
Owner SHENZHEN COOCAA NETWORK TECH CO LTD