Data deduplication method and device

A data and data structure technology, applied in the computer field, can solve problems such as resource waste and system resource waste, and achieve the effect of saving system resources

Active Publication Date: 2015-07-15
RUN TECH CO LTD BEIJING
View PDF5 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0013] 1. The same data will be saved in h1 and h2 at the same time, resulting in a waste of resources;
[0014] 2

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data deduplication method and device
  • Data deduplication method and device
  • Data deduplication method and device

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0034] figure 1 It is a flow chart of a data deduplication method provided by the first embodiment of the present invention picture , like figure 1 shown, including the following steps:

[0035] Step 101. Send a data collection request to a collection device, so that the collection device collects data from a network, and the data is network data packets or communication signaling.

[0036] Wherein, the data collection request includes a target address of the collected data, such as an IP address, and the collection device collects data from the network according to the target address. Wherein, the collected data may be network data packets, or communication signaling, or data files.

[0037] Wherein, the data collection request may contain multiple different target addresses, and the collection device sequentially acquires data from different networks according to the target addresses.

[0038] Step 102. Receive the first data sent by the collection device.

[00...

no. 2 example

[0064] figure 2 It is a flow chart of a data deduplication method provided by the second embodiment of the present invention picture , like figure 2 shown, including the following steps:

[0065] Step 201, receiving a deduplication request message input by a user.

[0066] For details, refer to the relevant description of this step in the foregoing embodiments, and details are not repeated here.

[0067] Step 202: Send a data collection request to the collection device, so that the collection device collects data from the network, and the data is network data packets or communication signaling.

[0068] Step 203. Receive the first data sent by the collection device.

[0069] For details, refer to the relevant description of this step in the foregoing embodiments, and details are not repeated here.

[0070] Step 204, detecting whether the first data is stored in the first data structure in the cache.

[0071] Wherein, the first data structure is used to store data ...

no. 3 example

[0094] image 3 It is a flow diagram of a data deduplication method provided by the third embodiment of the present invention picture , on the basis of the above-mentioned embodiments, this embodiment further adds the related steps of cleaning up the second data stored in the second data structure whose arrival time exceeds the preset duration, like image 3 As shown, it specifically includes the following steps:

[0095] Step 301, query the time when the data stored in the second data structure arrives in the cache.

[0096] For example, the time when the data stored in the second data structure arrives in the cache can be queried regularly, for example, queried every first preset time period. The first preset duration may be the preset duration, or any duration greater than the preset duration.

[0097] Step 302: Determine the second data that stays in the cache longer than a preset duration according to the time when the data arrives in the cache.

[0098] For exam...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a data deduplication method and a device. The method comprises the following steps of sending a data acquisition request to an acquisition device, so that the acquisition device acquires data from the network, wherein the data are network data packets or communication signaling; receiving first data sent by the acquisition device; detecting whether the first data exist in a buffer memory or not, if yes, discarding the first data, and if not, inserting the first data into the buffer memory. According to the embodiment of the invention, deduplication can be finished by storing one portion of data, so that not only can the aim of data deduplication be achieved, but also the system resource is saved.

Description

technical field [0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data deduplication method and device. Background technique [0002] With the development of computer and communication technology, the application of network has become popular rapidly, and it has increasingly become an indispensable tool for survival. At the same time, in order to meet the needs of network security and services, it is necessary to collect and analyze network data. Due to the network design and collection scheme, the collected data often has a large amount of duplicate data, which has a significant impact on subsequent storage and analysis. Therefore, in practical applications, data will be deduplicated before storage and analysis. [0003] The commonly used data deduplication method in the prior art is the double hash method. In the deduplication process of the double hash method, it mainly includes a data processing flow and an ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 陶小龙
Owner RUN TECH CO LTD BEIJING
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap