Unlock instant, AI-driven research and patent intelligence for your innovation.

Streaming data batch processing method and device

A streaming data and batch processing technology, applied in the direction of electrical digital data processing, special data processing applications, database management systems, etc., can solve data warehouse performance problems and other issues

Pending Publication Date: 2021-11-30
CHINA CONSTRUCTION BANK
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] At present, the existing technology can store streaming data in the data warehouse, which requires frequent write operations, which can easily lead to performance problems in the data warehouse

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Streaming data batch processing method and device
  • Streaming data batch processing method and device
  • Streaming data batch processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0065] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0066] Such as figure 1 As shown, this embodiment proposes the first stream data batch processing method, which may include the following steps:

[0067] S101. From the target storage space, determine the initial message position of the currently unconsumed message in the message to be consumed partition, and the messages stored in the message to be consumed partition are streaming data;

[0068] Wherein, the target storage space may be used a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a streaming data batch processing method and a streaming data batch processing device, which can consume at least part of messages in a to-be-consumed message partition within the maximum consumption duration, and uniformly warehouse the messages consumed within the maximum consumption duration. In this way, real-time or quasi-real-time processing of at least part of streaming data in the to-be-consumed message partition can be achieved, multiple insertion operations do not need to be executed on the target data warehouse in the warehousing operation process, and the abnormal probability of the target data warehouse is effectively reduced.

Description

technical field [0001] The invention relates to the technical field of streaming data processing, in particular to a batch processing method and device for streaming data. Background technique [0002] With the development of data processing technology, streaming data processing technology continues to improve. [0003] Among them, streaming data is a series of dynamic data collections with unlimited time distribution and quantity. Common streaming data processing scenarios include real-time recommendation and business monitoring. In such scenarios, the value of streaming data will decrease with the passage of time, and real-time or quasi-real-time processing of streaming data is required, and the streaming data can be quickly stored in a designated data warehouse for possible business queries Respond quickly. [0004] Currently, in the existing technology, frequent write operations are required in the process of storing streaming data to the data warehouse, which may eas...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/172G06F16/22G06F16/25
Inventor 赵瑞
Owner CHINA CONSTRUCTION BANK