Unlock instant, AI-driven research and patent intelligence for your innovation.

Kafka cluster data consistency guarantee method based on message heat

A consistent and hot technology, applied in digital data protection, electrical digital data processing, inter-program communication, etc., can solve problems such as the increase of operating costs, achieve the effect of balancing availability and performance, and ensuring reliability

Inactive Publication Date: 2018-02-06
CHONGQING UNIV OF POSTS & TELECOMM
View PDF4 Cites 14 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, because the consistency level is continuous and the classification is too fine, the continuous conversion of the consistency level causes a sharp increase in operating costs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Kafka cluster data consistency guarantee method based on message heat
  • Kafka cluster data consistency guarantee method based on message heat
  • Kafka cluster data consistency guarantee method based on message heat

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] The basic idea of ​​the present invention is: predict the future access popularity of the message according to the historical access frequency and heat prediction model of the message, then calculate the update frequency of the copy according to the established message partition copy, and finally determine the update of the copy according to the adaptive update model mechanism. This algorithm can better balance replica data consistency and cluster efficiency.

[0037] Such as figure 1 , figure 2 Shown are two scenarios of the number of real message visits.

[0038] The platform consists of three physical machines, three of which are equipped with Kafka and zookeeper. The specific configuration of each machine is shown in the following table:

[0039] List of hardware configurations

[0040]

[0041] The ubuntu operating syst...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a kafka cluster data consistency guarantee method based on message heat and belongs to the data storage and message middleware field. The method comprises steps that S1, a computer programming function partition heat model is established; S2, a heat pre-estimation model is established; S3, update frequency of a duplicate partition is calculated; and S4, the duplicate adaptive consistency strategy is established. The method is advantaged in that advantages of reliability of a kafka message queue and the adaptive synchronization strategy of a cloud storage system are respectively integrated, a kafka cluster duplicate data synchronization self-adaptive algorithm based on cluster message access heat is proposed, an LRU algorithm and an LFU algorithm are integrated during file heat prediction, the access time and the access frequency are both considered, so the prediction result is made to be more consistent with an actual value; data reliability is guaranteed, cluster network cost, memory and CPU extra source cost are all reduced, and system availability and performance are effectively balanced.

Description

technical field [0001] The invention belongs to the field of data storage and message middleware, and relates to a method for ensuring data consistency of a kafka cluster based on message heat. Background technique [0002] In recent years, with the continuous development of the Internet industry, the amount of data in various businesses has continued to increase. Under the environment of big data processing, such as stock trading, real-time weather forecast, network operation and maintenance monitoring, e-commerce real-time recommendation, etc. have promoted storm, Deployment and application of real-time computing frameworks such as s4 and sparkstreaming. These applications have higher and higher requirements for the horizontal expansion of the business system and data reliability. At the same time, the system has continuously improved requirements for asynchronous communication, high throughput and real-time data processing. Therefore, the emergence of Kafka is a matter o...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08G06F21/64G06F9/54
CPCH04L67/1095G06F21/64G06F9/546G06F2209/548H04L67/56
Inventor 蒋溢郭宗怀熊安萍祝清意詹妮
Owner CHONGQING UNIV OF POSTS & TELECOMM