Unlock instant, AI-driven research and patent intelligence for your innovation.

An event processing method and device in a Spark platform

An event processing and event technology, applied in the field of data processing, can solve problems such as inability to analyze data flow correlation, infer new data, etc.

Inactive Publication Date: 2019-03-15
NEW H3C BIG DATA TECH CO LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the object of the present invention is to provide a method and device for event processing in the Spark platform, to alleviate the existing real-time computing framework Spark Steaming cannot perform correlation analysis on data streams to infer new data from data streams question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • An event processing method and device in a Spark platform
  • An event processing method and device in a Spark platform
  • An event processing method and device in a Spark platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0028] According to an embodiment of the present invention, an embodiment of an event processing method in a Spark platform is provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

[0029] figure 1 It is a flowchart of an event processing method in a Spark platform according to an embodiment of the present invention, the method is applied to a Spark Streaming node, and a correlation engine is integrated in the Spark Streaming node, such as figure 1 As shown, the method includes the following steps:

[0030] Step S102, extracting the event entity from the message queue;

[0031] Step S104, processing the event entity into structured data;

[0032] Step S106, sending the ...

Embodiment 2

[0075]The embodiment of the present invention also provides an event processing device in the Spark platform. The event processing device in the Spark platform is mainly used to execute the event processing method in the Spark platform provided by the above-mentioned content of the embodiment of the present invention. The following provides the embodiment of the present invention The event processing device in the Spark platform will be introduced in detail.

[0076] Image 6 is a schematic diagram of an event processing device in a Spark platform according to an embodiment of the present invention, such as Image 6 As shown, the event processing device in the Spark platform mainly includes an extraction unit 10, a data conversion unit 20, a sending unit 30 and a correlation engine 40, wherein:

[0077] An extracting unit 10, configured to extract event entities from the message queue;

[0078] A data conversion unit 20, configured to process the event entity into structured...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an event processing method and device in a Spark platform, and relates to the technical field of data processing, and the method comprises the steps: extracting an event entityfrom a message queue; Processing the event entity into structured data; Sending the event entity processed into the structured data to an association engine; And the association engine writes the received event entities into the corresponding event classes, performs association calculation on the event entities in the event classes according to association rules, and outputs complex events. The technical problem that an existing real-time computing framework Spark Steamg cannot perform correlation analysis on the data flow so as to infer new data from the data flow is solved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to an event processing method and device in the Spark platform. Background technique [0002] The real-time computing framework Spark Steaming is an extension of Spark API (Application Programming Interface, application programming interface). The real-time computing framework Spark Steaming cannot process events one by one like the streaming processing framework Storm. The real-time computing framework Spark Streaming decomposes streaming computing into a series of short batch jobs, and the processing engine is Spark Core. Spark Core generates discrete data streams (DStream, Discretized Stream) from the input data of the real-time computing framework SparkStreaming according to the batch size. [0003] However, in the existing technology, the real-time computing framework Spark Steaming cannot obtain more valuable events from the discrete data stream DStream. Obt...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/54
CPCG06F9/546
Inventor 韩东亮龚红祥
Owner NEW H3C BIG DATA TECH CO LTD