Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data flow system resource prediction mechanism based on a deep belief network

A data flow system and resource prediction technology, applied in the direction of resource allocation, multiprogramming device, biological neural network model, etc., can solve the problem of resource load situation and node load resource shortage in which the system state cannot be predicted for a long time or a short time.

Inactive Publication Date: 2019-06-11
CHINA UNIV OF PETROLEUM (EAST CHINA)
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

For example, the log collection system Flume also has a lack of node load resources, and it is impossible to predict the system status of a certain period of time and the long-term or short-term resource load.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data flow system resource prediction mechanism based on a deep belief network
  • Data flow system resource prediction mechanism based on a deep belief network
  • Data flow system resource prediction mechanism based on a deep belief network

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0014] The present invention is based on the deep belief network (DBN) data flow system resource prediction mechanism predictor such as figure 1 As shown, the mechanism includes two stages of data preprocessing and model training.

[0015] Data preprocessing:

[0016] The neural network can play a good role in predicting nonlinear data, but the predictive effect will be weakened for data with strong linear factors. Preprocess the data with a smoothing method...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data flow system resource prediction mechanism based on a deep belief network (DBN), which is realized by adopting unsupervised deep learning and carrying out time sequence analysis on the basis of the DBN. The mechanism is divided into two stages, namely a data preprocessing stage and a model training stage: the data preprocessing stage mainly adopts a smoothing method topreprocess data, and a differential transformation is generated from original time sequence data to eliminate linear factors in the data; In the model training stage, a DBN network model composed oftwo layers of RBMs is used for carrying out fitting training on data, and then prediction analysis is carried out on node loads. According to the deep learning mechanism, prediction of short-term loadconditions and long-term load conditions is provided, and the state of the system can be comprehensively known when calculation tasks are submitted.

Description

technical field [0001] The invention relates to the fields of deep learning, time series analysis, and stream data processing, and in particular to a data stream system resource prediction mechanism based on a deep belief network. Background technique [0002] The log collection system of data flow usually faces the problem of lack of system resources. How to schedule system resources to achieve reasonable allocation is an important guarantee for system efficiency. For example, the log collection system Flume also has a lack of node load resources, and it is impossible to predict the system status in a certain period of time and the long-term or short-term resource load. As an unsupervised deep learning network model, the deep belief network can realize the automatic training of the model, use time series data as training, and predict the resource load. The technologies closest to the present invention in recent years are: [0003] (1) Deep Belief Network: DBN is a probabi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/50G06N3/063G06N3/08
Inventor 任鹏程张卫山房凯
Owner CHINA UNIV OF PETROLEUM (EAST CHINA)
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com