Data processing method, device, computer equipment and storage medium for stream computing

A data processing and stream computing technology, applied in the field of data processing, can solve the problem of unreliable calculation results of stream processing, and achieve the effect of solving the unreliable calculation results.

Active Publication Date: 2021-02-26
太平金融科技服务(上海)有限公司
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Based on this, it is necessary to provide a stream computing data processing method, device, computer equipment, and storage medium that can solve the problem of data duplication to solve the above-mentioned problem of unreliable calculation results of stream processing due to data duplication

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device, computer equipment and storage medium for stream computing
  • Data processing method, device, computer equipment and storage medium for stream computing
  • Data processing method, device, computer equipment and storage medium for stream computing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0043] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

[0044] The data processing method of stream computing provided by this application can be applied to such as figure 1 shown in the application environment. Among them, the stream processing engine 102 communicates with upstream systems (including source data systems that generate data, CDC software or log collection software, and message queues, etc.) 104 and downstream systems (including business systems such as APIs and class libraries at different levels) 106 through the network. communication. Wherein, the stream processing engine 102 may be, but not limited to, vari...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present application relates to a data processing method, device, computer equipment and storage medium. The method collects the data to be processed, extracts the key value from the data to be processed, uses the set hash rule to hash the key value, obtains the corresponding target hash value, and maps the target hash value to the pre-set hash value. The corresponding bit in the created bit array, if there is no data in the corresponding bit in the bit array, it means that the data to be processed is new data, so the data to be processed is processed according to the data processing request of the business system, and the processing is returned to the business system The target data obtained after processing the data is processed, because only the new data is processed, thus solving the problem of unreliable calculation results of stream processing due to data duplication.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a data processing method, device, computer equipment and storage medium for stream computing. Background technique [0002] With the development of data processing technology, there is a demand for real-time processing of massive and high-speed data in more and more fields. Since such requirements often exceed the capabilities of traditional data processing technology, based on this, the distributed stream processing model should be used And born. For example, Flink (distributed streaming data flow engine), as a new generation of distributed streaming processing engine, pulls data from upstream systems (such as CDC software, message queues or file collection systems), and requests data based on different levels of API or class libraries. It is processed to achieve high throughput and low latency, and can be applied to various complex real-time scenarios. ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/215G06F16/2455G06F16/22
CPCG06F16/215G06F16/22G06F16/24568
Inventor 邓才应万仁俊撒银利
Owner 太平金融科技服务(上海)有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap