Batch processing task triggering method and system, and computer-readable storage medium

A data processing system and batch processing technology, applied in the field of big data, can solve the problems of low timeliness, long offline computing delay, delay of 1 day, 1 week or one month, etc., to achieve the effect of shortening the delay

Pending Publication Date: 2021-11-19
ZHEJIANG DAHUA TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] At present, big data technology has become more and more mature, and the framework of real-time computing and offline computing has also tended to be mature. The result of real-time calculation is inaccurate; offline calculation is just the opposite, the timeliness is very low, and some calculations are even delayed by 1 day, 1 week or a month, and some problems such as out-of-order or repeated sending of data can be processed within this buffer time. The accuracy is high, so stream-batch integration has become the development direction, but the current stream-batch integration has the problems of isolation between real-time calculation and offline calculation and long delay of offline calculation

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Batch processing task triggering method and system, and computer-readable storage medium
  • Batch processing task triggering method and system, and computer-readable storage medium
  • Batch processing task triggering method and system, and computer-readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0016] Currently, the mainstream stream-batch integration solutions are the Lambda architecture and the Kappa architecture. The Lambda architecture is as follows: figure 1 As shown, the direct consumption data of the real-time branch produces real-time and inaccurate results, and the consumption data of the offline branch is persisted to the database, waiting for the trigger of the scheduled offline task to produce more accurate results, and then correcting the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a batch processing task triggering method and system, and a computer-readable storage medium. The data processing system is provided with a data processing architecture, and the data processing architecture comprises: a real-time processing layer and a batch processing layer; the real-time processing layer is used for receiving first to-be-processed data in real time, copying the first to-be-processed data to obtain second to-be-processed data, determining whether all the second to-be-processed data corresponding to a current batch processing task are received or not, and if all the second to-be-processed data corresponding to the current batch processing task are received, generating an event notification message; the batch processing layer is connected with the real-time processing layer and is used for processing all the received second to-be-processed data after receiving the event notification message so as to execute the current batch processing task. In this way, the delay time of batch processing tasks can be shortened.

Description

technical field [0001] The present application relates to the technical field of big data, and in particular to a batch processing task triggering method, system and computer-readable storage medium. Background technique [0002] At present, big data technology has become more and more mature, and the framework of real-time computing and offline computing has also tended to be mature. The result of real-time calculation is inaccurate; offline calculation is just the opposite, the timeliness is very low, and some calculations are even delayed by 1 day, 1 week or a month, and some problems such as out-of-order or repeated sending of data can be processed within this buffer time. The accuracy is high, so stream-batch integration has become the development direction, but the current stream-batch integration has the problems of isolation between real-time calculation and offline calculation and long delay of offline calculation. Contents of the invention [0003] The present a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/54G06F9/48
Inventor 徐伟平周明伟刘金山王亮
Owner ZHEJIANG DAHUA TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap