A Distributed Environment Coordinated Consumption Queue Method and Device

A distributed environment and queue technology, which is applied in the field of distributed environment coordinated consumption queues and distributed environment coordinated consumption queue devices, can solve problems such as inaccurate consumption results and repeated data consumption

Active Publication Date: 2021-04-13
ALIBABA GRP HLDG LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although the Kinesis Client Library supports new clients to consume data, when performing operations such as load balancing, a client directly re-consumes the preempted queue fragments, which will lead to repeated consumption of some data, resulting in inconsistent consumption results. accurate

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Distributed Environment Coordinated Consumption Queue Method and Device
  • A Distributed Environment Coordinated Consumption Queue Method and Device
  • A Distributed Environment Coordinated Consumption Queue Method and Device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0034] refer to figure 1 , which shows a flow chart of the steps of an embodiment of a method for coordinating a consumption queue in a distributed environment of the present application, which may specifically include the following steps:

[0035] Step 110, for a queue segment to be consumed, the status data of the queue segment obtained based on the lease agreement;

[0036] The embodiment of this application provides a set of leasing protocols for each shard in the multi-consumer cooperative consumption queue in a distributed environment, and realizes the process of multi-computing nodes cooperatively consuming data in multiple shards.

[0037]According to the lease agreement, when a client consumes a shard, it will record its consumption progress in a persistent storage space and update the state data of the shard. The status data of the shard is used to determine whether the shard is being consumed by clients.

[0038] Then for the current client worker A, if it preempt...

Embodiment 2

[0288] refer to figure 2 , which shows a structural block diagram of a system embodiment of a distributed environment coordinated consumption queue of the present application, which may specifically include the following modules:

[0289] A status data acquisition module 310, configured to obtain status data of the queue segment based on a lease agreement for a queue segment to be consumed;

[0290] A consumption judging module 320, configured to judge whether other clients are consuming the queue fragments according to the status data;

[0291]A progress acquiring module 330, configured to update the state data of the queue slice and obtain the current consumption progress of the queue slice if it is determined that no other client is consuming the queue slice;

[0292] The consumption module 340 is configured to continue to consume the queue fragments according to the current consumption progress, and record the new consumption progress of the queue fragments.

[0293] In...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

Embodiments of the present application provide a method and device for coordinating consumption queues in a distributed environment, which relate to the field of distributed technologies. The method includes: for a queue fragment to be consumed, based on the state data of the queue fragment acquired by the lease agreement; according to the state data, judging whether other clients are consuming the queue fragment; if determined If no other client is consuming the queue fragment, update the state data of the queue fragment, and obtain the current consumption progress of the queue fragment; continue to consume the queue fragment according to the current consumption progress, and set The new consumption progress of the queue fragments is recorded. This application can seamlessly transmit the consumption progress of the queue fragment to the current client when the current client A preempts the queue fragments that other clients are consuming under the circumstances of load balancing, a certain client crash, etc. Avoid repeated consumption of some data and make consumption results more accurate.

Description

technical field [0001] The present application relates to the field of distributed technologies, in particular to a distributed environment coordinated consumption queue method and a distributed environment coordinated consumption queue device. Background technique [0002] With the advent of the era of cloud computing and big data, the sources of data are getting wider and faster, and the quantity is getting bigger and bigger. For example, on the web server, various clients, sensors, etc., a large amount of data is generated in real time, and various user access requests, monitoring data, program running status and other information are recorded. In order to better mine the value of data, there are often multiple systems consuming these data, such as various real-time and offline systems for user behavior analysis, monitoring and alarming, program intrusion detection, etc. [0003] In order to reduce the coupling between data production and consumption, a queue system is u...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50
CPCG06F9/5061G06F9/5083G06F9/50H04L65/40
Inventor 孙廷韬
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap