Data fragment synchronization method based on message queue and system thereof

A technology of data fragmentation and message queue, applied in digital transmission system, transmission system, electronic digital data processing and other directions, can solve the problems of data node malicious data, data node failure, inconsistency, etc. The effect of ensuring orderliness

Pending Publication Date: 2020-11-24
中国科学院计算技术研究所数字经济产业研究院 +1
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to provide a method and system for synchronizing data slices based on mes

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data fragment synchronization method based on message queue and system thereof
  • Data fragment synchronization method based on message queue and system thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] like figure 2 As shown, the present embodiment is a message queue-based data fragmentation synchronization system, which has a write node, a message queue, a management node and several data nodes, wherein the write node writes the data to be stored into the message queue; the data node Listen to the data write messages in the message queue, and write the data into the corresponding data shards according to the pre-established data sharding rules; when the cluster needs to synchronize the data shards, the management node sends an instruction message through the message queue , to coordinate the data nodes in the entire cluster to synchronize data shards.

[0040] In this example, when the synchronization system has a new data node joining the cluster, the data fragmentation synchronization method is as follows:

[0041] S0-1. The newly added data node sends a PING message to the management node.

[0042] S0-2. After receiving the PING message, the management node con...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data fragment synchronization method based on a message queue and a system thereof. The invention aims to provide the data fragment synchronization method based on a messagequeue and the system to solve the problem of data inconsistency caused by data node failure and data node disgusting in the data synchronization process. According to the technical scheme, the methodcomprises the steps that S1, a management node obtains a message sequence number M1, sent by each data node, of a consumed message queue of the current data node; S2, the management node takes the maximum message serial number in the message serial numbers M1 of the consumed message queues of the data nodes as a selected message serial number N; S3, when N is greater than M1, the data node continues to consume the message queue, and consumption is stopped until the message serial number of the message queue consumed locally is N; and S4, the management node acquires the message serial numberM2 of the consumed message queue of the current data node sent by each data node and the HASH value of each data fragment held by the current data node. The method is suitable for the field of data storage.

Description

technical field [0001] The invention relates to a method and system for synchronizing data slices based on message queues. Applicable to the field of data storage. Background technique [0002] Data nodes form a distributed data storage system (see figure 1 ), in order to increase the availability of the system, the data is stored in data shards in the data nodes, and the data shards are redundantly stored in the system. For each data shard with the same number, there will be multiple copies in the system, and the data stored in each data shard copy should be exactly the same. [0003] When a new data node joins or exits the cluster, the data fragments in the data node need to be synchronized. When a new node joins, the purpose of redistribution is to make the existing data fragments more evenly distributed in each node; When a node exits, the purpose of reallocation is to keep enough copies of the data shards in the cluster. [0004] However, the existing technology onl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/08G06F3/06
CPCH04L67/1095G06F3/062G06F3/0659G06F3/067H04L9/50Y02D30/50
Inventor 尚德重焦臻桢丁慧田锐
Owner 中国科学院计算技术研究所数字经济产业研究院
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap