Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and device for processing stream data

A stream processing and stream data technology, applied in the field of data processing, can solve the problems of stream data processing delay, long time, and inability to process stream data in real time.

Active Publication Date: 2014-12-17
HUAWEI CLOUD COMPUTING TECH CO LTD
View PDF5 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In the above method, there is a certain time interval between stopping the first stream processing model and starting the second stream processing model. During this time interval, the stream processing platform cannot process the stream data in real time.
In addition, if the first stream processing model includes many stream operators, and the data cached in the stream window corresponding to the stream operators is also large, at this time, the state of each stream operator at the current moment and each The data cached in the stream window corresponding to the stream operator takes a long time, which in turn causes a delay in processing the stream data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for processing stream data
  • Method and device for processing stream data
  • Method and device for processing stream data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0063] figure 1 It is a flowchart of a method for processing streaming data provided by an embodiment of the present invention. see figure 1 , the method includes:

[0064] Step 101: When the first processing operation needs to be performed on the stream data generated by the business, the first stream processing model is received. The first stream processing model includes the stream operator corresponding to the first processing operation. The first processing operation is the original A new processing operation besides the processing operation.

[0065] Step 102: Modify the first stream processing model according to the second stream processing model to obtain a third stream processing model, the second stream processing model is a stream processing model running on the stream processing platform for processing the stream data, and The second stream processing model includes legacy processing operations.

[0066] Step 103: Stop the first stream merging model when a stop...

Embodiment 2

[0087] figure 2 It is a flowchart of a method for processing streaming data provided by an embodiment of the present invention. see figure 2 , the method includes:

[0088] Step 201: Receive a fourth stream processing model.

[0089] Specifically, the dynamic scheduling module receives the fourth stream processing model. Wherein, the dynamic scheduling module may be a module in the stream processing platform, or may be a module in other separate devices. Preferably, the dynamic scheduling module is a module in the stream processing platform. This embodiment of the present invention does not specifically limit it.

[0090] Among them, when processing the stream data generated by the business, developers can develop the fourth stream processing model according to the required processing results, the fourth stream processing model is used to process the stream data of the business, and the fourth stream processing model includes multiple stream operators, and each stream o...

Embodiment 3

[0128] Figure 9 It is a schematic structural diagram of a device for processing streaming data provided by an embodiment of the present invention. see Figure 9 , the device includes: a first receiving module 901, a first modifying module 902, a stopping module 903, a second modifying module 904 and a starting module 905;

[0129] The first receiving module 901 is configured to receive the first stream processing model when the first processing operation is performed on the stream data generated by the service, the first stream processing model includes the stream operator corresponding to the first processing operation, and the first processing operation is in the New processing operations other than the original processing operations of streaming data;

[0130] The first modifying module 902 is configured to modify the first stream processing model according to the second stream processing model to obtain a third stream processing model, and the second stream processing m...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and a device for processing stream data, and belongs to field of data processing. The method comprises the following steps: when first processing operation is performed on stream data generated by a business, receiving a first stream processing model, wherein the first stream processing module comprises a stream operator which corresponds to the first processing operation; modifying the first stream processing model according a second stream processing module to obtain a third stream processing module, wherein the second stream processing module is a stream processing module which runs in a stream processing platform and is used for processing stream data; when a stop instruction is received, stopping a first stream combining model; modifying the first stream combining model to obtain a second stream combining model; when a start instruction is received, starting the second stream combining model, and processing the stream data through the second stream processing model, the third stream processing model and the second stream combining module. By adopting the method and the device, the stream operator can be increased under the condition of not influencing the result of original processing operation, and delay in processing of the stream data is avoided.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method and device for processing streaming data. Background technique [0002] Stream processing technology uses a distributed computing architecture and processes stream data generated by business in real time through a preset stream processing model. Moreover, the stream processing technology is generally carried on the stream processing platform, and the stream processing platform can process stream data based on multiple stream operators included in the preset stream processing model. Among them, the stream processing model is used to process, analyze and process stream data, and stream operators are used to complete a type of processing operation on stream data. When additional analysis and processing of stream data is required, it is necessary to On the premise that the operator processes the result, add a stream operator to the stream processing model, and process the str...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/46
CPCG06F9/46
Inventor 戴劲张克星何志强王澜郑阳平智伟
Owner HUAWEI CLOUD COMPUTING TECH CO LTD