System and method for dynamically setting batch intervals under disperse flow processing model

A technology for processing models and discrete flows, which is used in electrical digital data processing, multi-programming devices, program control design, etc.

Active Publication Date: 2016-06-15
SHANDONG UNIV
View PDF3 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, how to determine a reasonable batch division and make scalability adjustments according to resources has become a challenge

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for dynamically setting batch intervals under disperse flow processing model
  • System and method for dynamically setting batch intervals under disperse flow processing model
  • System and method for dynamically setting batch intervals under disperse flow processing model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0069] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0070] 1.1 Batch interval and delay

[0071] Batch processing completes a series of deterministic calculations in a specific cycle, and defines this specific cycle as a batch (Batch). The size of its period is defined as the batch interval length (intervalsize). A time period can be defined in the system as the interval length. For example, if the data stream is read from the data source in units of 1s, then 1s is the interval length. In addition, a certain amount of data size can also be used as the interval length (interval size). For example, the data source reads stream data in units of 1 Mb, then 1 Mb is the interval length. For the batch-based data flow system, the present invention uses the time period as the interval length instead of the data size as the interval length. This is because the different rates of different data sets lead to diff...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a system and method for dynamically setting a batch interval under a disperse flow processing model.The system comprises a main node and an observation module, wherein the main node is connected with a data source and a plurality of working nodes respectively, a receiver and a task management module are deployed on the main node, the receiver is connected with the working nodes sequentially through a batch control module and the observation module, and the main node obtains flow data from the data source.The observation module is used for monitoring load information and network flow information in clusters and calculating a batch interval value meeting the set conditions.The batch control module transmits the batch interval value to the receiver and segments batches for the flow data in the receiver according to the batch interval value.The task management module divides the batches into distributed tasks and allocates the distributed tasks to the working nodes for calculation.The batch interval value refers to the time interval of unprocessed data obtained from the receiver.The data processed according to the batch interval value is regarded as the batch corresponding to the interval value.

Description

technical field [0001] The invention relates to a system and method for dynamically setting batch interval size under a discrete stream processing model. Background technique [0002] In recent years, people's demand for analysis and processing of massive data is increasing. Especially with the deep understanding of the real-time and effectiveness of data, the real-time processing technology of "big data" has increasingly penetrated into various fields of economic development, social progress and human life, and has become an active factor in productivity. For example, in twitter or Sina Weibo, it is necessary to calculate the latest hot topics in real time; the recommendation system in the e-commerce website needs to analyze the purchase behavior of users, and then push the products in real time to increase sales. These application scenarios have high requirements for real-time computing and require the system to have certain fault tolerance. [0003] The traditional dist...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
CPCG06F9/5083G06F2209/5017
Inventor 史玉良崔立真李庆忠郑永清刘金
Owner SHANDONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap