Multi-thread data processing method, device and medium based on streaming computing framework

A streaming computing and data processing technology, applied in the field of big data processing, can solve the problem of inaccurate calculation results

Active Publication Date: 2021-03-16
浙江岩华文化科技有限公司
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The embodiment of the present application provides a multi-threaded data processing method, electronic device, and storage medium based on a streaming computing framework, so as to at least solve the problem of inaccurate calculation results based on a streaming computing framework in the related art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-thread data processing method, device and medium based on streaming computing framework
  • Multi-thread data processing method, device and medium based on streaming computing framework
  • Multi-thread data processing method, device and medium based on streaming computing framework

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application. Based on the embodiments provided in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application. In addition, it can also be understood that although such development efforts may be complex and lengthy, for those of ordinary skill in the art relevant to the content disclosed in this application, the technology disclosed in this application Some design, manufacturing or production changes based on the content are just conventional technical means, and should not be under...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present application relates to a multi-threaded data processing method, device and storage medium based on a streaming computing framework. The first processing result is obtained by triggering the target window to process data slices based on the preset first water level; if the unprocessed data If there is a lag data slice whose event timestamp is less than the end timestamp of the target window in the slice, record the system timestamp of the streaming computing framework receiving the lag data slice; set the second watermark, where the delay time of the second watermark is not less than The difference between the system timestamp of the hysteresis data slice and the end timestamp of the target window; retrigger the target window to process the data slice based on the second watermark, obtain the second processing result, and correct the first processing result according to the second processing result, solving the problem Based on the inaccurate calculation results of the streaming computing framework, the accuracy of the calculation results is improved while ensuring a certain timeliness of data processing.

Description

technical field [0001] The present application relates to the technical field of big data processing, in particular to a multi-threaded data processing method based on a streaming computing framework, an electronic device and a storage medium. Background technique [0002] Usually in a big data system, using Flink to process real-time data is a better choice. Flink is a streaming computing framework for stateful computing on unbounded and bounded data streams. It can run in a cluster environment and perform computations on real-time data at memory execution speed and at any scale. There are three time concepts in Flink, namely event time stamp (Event Time), intake time stamp (Ingestion Time) and system time stamp (Processing Time). The event timestamp represents the time generated in the objective world. For example, a log is generated when a user clicks on a web page; the ingestion timestamp represents the time when the log reaches Flink; the system timestamp represents th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/38
CPCG06F9/3867
Inventor 王秀双马旭东赵玉红李良敏赵春波
Owner 浙江岩华文化科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap