Dynamic mining method for distributed data streaming

A data flow and distributed technology, applied in the field of big data processing, can solve the problems of unclear technical details, high cost, and little research, so as to reduce the possibility of data being discarded, reduce data discarding, and reduce network transmission costs Effect

Inactive Publication Date: 2019-04-23
FUJIAN UNIV OF TECH
View PDF1 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The former is simple, but the cost of network transmission is too high for large-scale, fast-flowing online big data
The latter obviously has efficiency advantages, but there are few studies, and naturally many technical details are still unclear

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Dynamic mining method for distributed data streaming
  • Dynamic mining method for distributed data streaming

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] Such as figure 1 Or shown in 2, the present invention discloses a kind of dynamic mining method of distributed data flow, and it comprises the following steps:

[0038] Step 1, each local node collects the current data block at the current time t, and performs micro-cluster processing on the current data block of each local node;

[0039] Step 2, each local node performs incremental micro-cluster update of the local mode: each local node incrementally updates the current data block processed by the micro-cluster collected at time t and the local mode maintained at time t-1 to form t local patterns of time;

[0040] Step 3, local mode transmission stage: upload the local mode of each local node at time t to the central node;

[0041] Step 4, the central node reconstructs the global sample data set based on micro-clusters after receiving the local patterns of all local nodes at time t:

[0042] Step 5: The central node performs a newly learned basic learner based on th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a dynamic mining method for distributed data streaming, which comprises the following steps of: S1, by each local node, collecting current data blocks at a current t moment andcarrying out micro cluster processing; S2, by each local node, carrying out incremental micro cluster updating of a local mode; S3, executing a local mode transmission stage, i.e., uploading the local mode of each local node at the t moment to a central node; S4, by the central node, after receiving the local modes of all the local nodes at the t moment, reconstructing a global sample data set onthe basis of a micro cluster; and S5, by the central node, executing a basic learner for new learning on the basis of the global sample data set, and carrying out incremental updating of a global mode in a current state on the basic learner for new learning. According to the invention, in a local mining mode, data can be processed locally to the greatest extent, so as to reduce the probability that the data is discarded.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a dynamic mining method for distributed data streams. Background technique [0002] Big data is a demand-driven concept. Although the 4V attributes of big data have been given, they still describe the appearance of big data, so it is still necessary to find a standardized data structure to accurately describe the technical characteristics of big data. Aiming at the technical characteristics of distributed and fluidity hidden in big data, a data structure called distributed data flow can provide an ideal way for the expression of certain types of big data. [0003] Distributed data flow captures the two main technical characteristics of big data, distributed and fluid, and can help solve the regularized analysis of many types of big data. For example: in a large-scale network monitoring system, it may be necessary to set up multiple monitoring sites to collect data du...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08H04L12/24G06K9/62
CPCH04L41/14H04L67/10H04L67/5651G06F18/23213
Inventor 毛国君
Owner FUJIAN UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap