Distributed database storage communication compression method based on dbDedup

A compression method and database technology, applied in database update, database distribution/replication, digital data information retrieval, etc., can solve problems such as underutilized space saving, avoid a lot of I/O overhead, high compression rate, improve memory The effect of efficiency

Pending Publication Date: 2020-10-30
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, doing so results in an underutilization of the possible space savings

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed database storage communication compression method based on dbDedup
  • Distributed database storage communication compression method based on dbDedup

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0064] The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

[0065] A distributed database storage and communication compression method based on dbDedup, the specific scheme realized by the method includes:

[0066] Such as figure 2 As shown, using a simple distributed database consisting of a client, a primary node, and a secondary node, where the primary node handles write requests and asynchronously pushes updates to the secondary nodes in the form of operation log (oplog) batches, The main operations of dbDedup for DBMS:

[0067] Insert: The master node writes a new record to its local database and appends the record to its oplog. Each oplog entry includes a timestamp and a payload containing the inserted record. When the size of unsynced oplog entries reaches a threshold, the primary node sends them to the secondary node in batches. The secondary node receives the update, appends the update to...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed database storage communication compression method based on dbDedup. The method is based on dbDedup, a good compression rate and a relatively low memory utilization rate are achieved at the same time by using a similarity-based dump; a new record is written into a local operation log in an insertion or update query period of a dedump encoding workflow used whenpreparing for updating record data for local storage and remote copying, and the new record is encoded into contents outside a critical path in the background through dbDedump. According to the method, the dbDedup is adopted to be combined with partial indexes and byte-level delta compression, so that the compression rate higher than that of block-level compression and block-based dedup can be achieved, and meanwhile, the memory efficiency can also be improved; a large number of I / O overhead involved in accessing delta encoded records is avoided by using novel encoding and caching mechanisms.

Description

technical field [0001] The present invention relates to the technical field of distributed database storage communication compression, and specifically provides a distributed database storage communication compression method based on dbDedup, which is used in a similarity-based deduplication (dedup) scheme for an online database management system, and is implemented based on dbDedup A device that reduces storage usage and the amount of data transferred for remote replication. Background technique [0002] With the popularity of web-based applications, more data is stored in various forms of databases today. Database compression was applied to this problem as data grew faster than hardware costs fell. For database storage, in addition to saving space, compression helps reduce the amount of disk I / O (input / output) and improves performance because the data queried fits on fewer pages. For distributed databases that are replicated across geographic regions, there is also a str...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/16G06F16/174G06F16/18G06F16/182G06F16/23G06F16/2455G06F16/27
CPCG06F16/162G06F16/1744G06F16/1815G06F16/182G06F16/2379G06F16/24552G06F16/27
Inventor 夏博涵王瀚墨陈磊
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap