Check patentability & draft patents in minutes with Patsnap Eureka AI!

Streaming calculation method and device, equipment and storage medium

A technology of streaming computing and computing tasks, applied in the computer field, can solve problems such as data asynchronousness of multiple real-time data sources, and achieve the effect of solving data asynchronous problems and high accuracy

Pending Publication Date: 2021-06-22
CHINA CITIC BANK
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Embodiments of the present invention provide a streaming computing method, device, device, and storage medium to solve the problem of data out-of-sync from multiple real-time data sources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Streaming calculation method and device, equipment and storage medium
  • Streaming calculation method and device, equipment and storage medium
  • Streaming calculation method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0035] figure 1 It is a flow chart of a streaming computing method provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of performing correlation calculation on real-time data from multiple real-time data sources. The method can be executed by a streaming computing device, and the device can be realized by software and / or hardware. see figure 1 , the streaming computing method provided by the embodiment of this application includes:

[0036] Start at least one worker process for the pending target streaming computing task.

[0037] Wherein, the target stream computing task is a stream computing task to be processed. A worker process refers to a process that processes target streaming computing tasks. A process is a running activity of a program in a computer on a certain data set, and is the basic unit for resource allocation and scheduling by the system

[0038] Controlling the at least one working process to compete on zooke...

Embodiment 2

[0053] figure 2 It is a flow chart of a streaming computing method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment, in order to solve the problem that after the process associated with the worker node is closed, the worker node has not successfully processed the delay queue takeover problem, the control of all work processes including the leader process in After registering as a worker node of the target streaming computing task on the leader management directory, the method further includes:

[0054] The leader node regularly checks whether all delay queues are managed by worker nodes;

[0055] If it is detected that there is no worker node processing in the target delay queue, then continue to check whether the target delay queue is empty;

[0056] If the target delay queue is empty, then clear the delay queue; otherwise, select a worker node to take over according to the load of the current worker node, and send a takeo...

Embodiment 3

[0067] image 3 It is a flow chart of a streaming computing method provided by Embodiment 3 of the present invention. This embodiment is based on the above embodiments, in order to solve the resource pressure problem caused by the excessively long delay queue on zookeeper, the control of all work processes including the leader process is registered on the leader management directory as After the worker node of the target streaming computing task, the method also includes:

[0068] The leader node checks the last modification time of each delay queue. If the last modification time of the queue exceeds the configured time threshold, it checks whether the queue is taken over by a worker node;

[0069] If there is no worker node management, the leader node cleans up the queue;

[0070] If there is a worker node management, the leader node issues a cleanup request for the queue, and the worker node associated with the queue accepts the request and is responsible for checking whet...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a streaming computing method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: judging whether data needs to be subjected to delay processing or not if detecting data which fails to be subjected to real-time correlation calculation in a streaming calculation process by each worker node; if delay processing needs to be carried out, registering the information of the data to a delay queue of the worker node associated with the data; according to the configured time period, enabling the delay processing thread to regularly and sequentially obtain data information from the delay queue and check the data information to confirm whether the data can be successfully processed or not; if the data can be successfully processed, processing the data and deleting the data node from the delay queue, otherwise, waiting for the processing of the next period. The embodiment of the invention provides a streaming computing method and device, equipment and a storage medium, and aims to solve the problem that data of a plurality of real-time data sources are asynchronous.

Description

technical field [0001] The embodiments of the present invention relate to the field of computer technologies, and in particular, to a streaming computing method, device, device, and storage medium. Background technique [0002] As modern businesses have higher and higher requirements for real-time data, streaming computing technology has received more and more attention from enterprise real-time architectures because of its real-time performance advantages. Although the current streaming data plays a very important role in real-time data requirements, there are also difficulties that are difficult to solve, that is, the problem of real-time data association: In traditional data processing, it is often unavoidable that data from multiple data sets needs to be associated with calculations scene. The most typical ones are multi-table JOIN operations common in relational databases or multi-table joint calculations under where conditions. [0003] When using streaming technolog...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48H04L29/08
CPCG06F9/4881G06F9/4843H04L67/10H04L67/60
Inventor 杨全文
Owner CHINA CITIC BANK
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com