Stream processing method and system based on massive real-time Internet dpi data
A stream processing and Internet technology, applied in the field of big data processing, can solve problems such as unsuitable for processing real-time massive data and large delay, and achieve real-time analysis and statistics, reduce delay, and improve throughput
Active Publication Date: 2020-03-31
江苏号百科技有限公司
View PDF5 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
However, Hadoop also has some shortcomings. It can only support offline data processing. Only when the data is written into Hadoop's local storage can further calculation and analysis be carried out. There is a large delay and it is not suitable for processing real-time massive data. , cannot meet and respond to some needs and businesses that are sensitive to data processing delays, so it is necessary to build a stream processing method that can process real-time data to meet real-time business needs
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0039] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.
[0040] Unless the context clearly states otherwise, the number of elements and components in the present invention can exist in a single form or in multiple forms, and the present invention is not limited thereto. Although the steps in the present invention are arranged with labels, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It can be understood that the term "and / or" used herein refers to and covers any and all possible combina...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a mass real-time internet DPI data-based flow processing method. The method comprises the steps of receiving mass real-time internet DPI data by an interface protocol layer, collecting, cleaning and filtering the mass real-time internet DPI data; receiving DPI data from the interface protocol layer by a Kafka cluster, and storing the received DPI data into the particular partition of a corresponding topics; acquiring DPI data from the topics of the Kafka cluster at the preset interval of a Storm cluster, conducting the corresponding pretreatment on the above data by the Topology of a corresponding processing unit, and outputting the resultant data of the pretreatment to the corresponding Topics of the Kafka cluster; acquiring the DPI data after the pretreatment of the Storm cluster from the Topics of the Kafka cluster at the preset time interval of a Spark Streaming cluster, copying and distributing the pre-treated DPI data, and storing an obtained final processing result in the database of a KV database cluster in the form of key value. The invention also provides a system of the mass real-time internet DPI data-based flow processing method.
Description
technical field [0001] The invention belongs to the technical field of big data processing, and in particular relates to a streaming processing method and system based on massive real-time Internet DPI data. Background technique [0002] In recent years, the speed of Internet development has grown rapidly, and the data on it has also continued to grow. Especially with the rise of the mobile Internet, diversified data has made our analysis and mining of various data more urgent. How to dig deep from these massive data and create greater and more useful value has always been the goal of the big data industry. [0003] At present, the mainstream big data processing methods are all based on Hadoop. The emergence of Hadoop makes it easier for people to analyze massive data. The MapReduce programming model on it can run and process in parallel on each node, and Hadoop has good reliability Scalability, nodes can join dynamically without affecting the normal operation of the cluste...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): H04L12/26H04L29/08
CPCH04L43/028H04L67/10
Inventor 黄凯翔周蓉张国华许睿
Owner 江苏号百科技有限公司