A method and system for sequential consumption of data

A technology of consumption data and data processing system, which is applied in the field of sequential consumption data, can solve problems such as the inability to guarantee the order of data production data, and achieve the effect of avoiding missing processing

Active Publication Date: 2020-08-07
北京明朝万达科技股份有限公司
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, in the case of multiple partitions, the consumer cannot guarantee that the data obtained from multiple partitions is in the order of the producer's production data. It can only ensure that in a partition of the same topic, the data consumed by the consumer is stored according to the data landing The order of the data in the current partition (this is why the consumed data is ordered when a topic has only one partition)

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for sequential consumption of data
  • A method and system for sequential consumption of data
  • A method and system for sequential consumption of data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the schematic embodiments and descriptions are only used to explain the present invention, but are not intended to limit the present invention.

[0039] MySql: A relational database management system, a relational database stores data in different tables instead of putting all the data in one big warehouse. It is a database built on the basis of the relational database model. It uses concepts and methods such as set algebra to process the data in the database. It is also a set of formally descriptive tables organized into a table. A special collection of data items in which data can be accessed or recalled in many different ways without reorganizing the database tables.

[0040] Kafka: An open source stream processing platform developed by the Apache Software Foundation, written in Scala and Java. Kafka is a high-throughput distributed...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and a system for sequential consumption data. A data producer client is connected to a data processing system and sends data to the data processing system. A producerthread of the data processing system stores the data into the database management system according to the subject of the received data and the partition number of the current data processing system. The consumer thread of the data processing system obtains the offset of the partition number corresponding to the current consumer ID according to the ID of the current consumer. The data consumer thread obtains data from the partition of the data processing system according to the offset of the partition number, matches and integrates the data stored in the database management system; the matchedand consolidated data results are returned to the corresponding data consumer client. The technical proposal of the invention solves the disorder problem of the consumption data under the condition ofmultiple partitions of the same subject of Kafka.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method and system for sequentially consuming data. Background technique [0002] As a popular relational database, MySql has high storage performance and supports ordered storage of data and query operations. As a popular distributed publish-subscribe message system, Kafka supports multiple producers to produce data messages to it, and supports multiple consumers to consume the produced data messages. [0003] When there is only one partition for the same topic in Kafka, consumers consume data sequentially during the process of consuming data. However, in the case of multiple partitions, the consumer cannot guarantee that the data obtained from multiple partitions is in the order of the producer's production data. It can only ensure that in a partition of the same topic, the data consumed by the consumer is stored according to the data landing The order of the data in the curre...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/28G06F16/2458
Inventor 董晓斌赵晶王志海喻波董爱华安鹏
Owner 北京明朝万达科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap