Eureka AIR delivers breakthrough ideas for toughest innovation challenges, trusted by R&D personnel around the world.

Data stream processing method and device, readable storage medium and processor

A processing method and technology of a processing device, applied in the field of data processing, can solve the problems of waste, the number of divisions cannot be enlarged indefinitely, and the orderliness of business cannot be guaranteed, so as to achieve the effect of fast processing and guaranteeing consistency.

Inactive Publication Date: 2020-01-21
KE COM (BEIJING) TECHNOLOGY CO LTD
View PDF11 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] Kafka or rockmq can be used as a message component, and the data can be divided into several parts by partitioning the same topic (topic), but each part of data can only be used by one consumer of one consumer group at the same time To deal with it, if you want to improve consumption efficiency, the conventional method can increase the number of consumers by adjusting the number of divisions, but the number of divisions cannot be enlarged infinitely; in addition, in general projects, the same consumer handles distribution through a single thread This is undoubtedly a waste. At the same time, even if a consumer adopts a multi-threaded method, it is relatively disorderly to consume the same divided data. In special cases, the order of the business cannot be guaranteed.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data stream processing method and device, readable storage medium and processor
  • Data stream processing method and device, readable storage medium and processor
  • Data stream processing method and device, readable storage medium and processor

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] The specific implementation manners of the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation manners described here are only used to illustrate and explain the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

[0025] figure 1 It is a flowchart of a data stream processing method provided by an embodiment of the present invention. Such as figure 1 As shown, the method includes:

[0026] Step S11, dividing the data stream into a plurality of divided data;

[0027] Specifically, kafka or rockmq can be used as a message component, and kafka can divide the data stream of the same topic (topic) into multiple parts to form multiple partitioned data.

[0028] Step S12, for at least one piece of divided data among the multiple pieces of divided data, configuring a data business key for each p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data stream processing method and device, a readable storage medium and a processor, and belongs to the technical field of data processing. The method comprises the following steps: dividing a data stream into a plurality of parts of division data; for at least one piece of division data in the plurality of pieces of division data, configuring a service primary key of the data for each piece of division data; according to the service main key, dividing the same type of data in each piece of divided data into the same memory queue to form a plurality of memory queues; and performing parallel processing on the plurality of memory queues. According to the data stream processing method and device, the readable storage medium and the processor, the data stream can be rapidly processed while the consistency of data services is ensured.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data stream processing method, device, readable storage medium and processor. Background technique [0002] Kafka or rockmq can be used as a message component, and the data can be divided into several parts by partitioning the same topic (topic), but each part of data can only be used by one consumer of one consumer group at the same time To deal with it, if you want to improve consumption efficiency, the conventional method can increase the number of consumers by adjusting the number of divisions, but the number of divisions cannot be enlarged infinitely; in addition, in general projects, the same consumer handles distribution through a single thread This is undoubtedly a waste. At the same time, even if a consumer adopts a multi-threaded method, the same partitioned data is consumed relatively disorderly. In special cases, the order of the business cannot be g...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/54G06F9/46
CPCG06F9/466G06F9/546G06F2209/548
Inventor 崔云鹏
Owner KE COM (BEIJING) TECHNOLOGY CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com