Unlock instant, AI-driven research and patent intelligence for your innovation.

A method of incremental calculation based on mapreduce

An incremental and incremental data technology, applied in the computer field, can solve the problems of lack of versatility, limited efficiency improvement, complex system, etc., to avoid repeated calculation and improve efficiency.

Active Publication Date: 2017-07-25
UNIV OF SCI & TECH OF CHINA
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At present, for the MapReduce computing model, the system improved on it is relatively complex, and the efficiency improvement is limited. Most of them are designed for a certain type of specific problem and lack versatility.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method of incremental calculation based on mapreduce
  • A method of incremental calculation based on mapreduce
  • A method of incremental calculation based on mapreduce

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0032] figure 1 A flow chart of a MapReduce-based incremental calculation method provided by the embodiment of the present invention; as figure 1 As shown, the method mainly includes the following steps:

[0033] Step 11. Create an incremental processing model for caching different historical processing results, including: a model for caching combiner results, a model for caching intermediate results, and a model for direct reuse of results;

[0034] Step 12. When the input data is obtained, select the corresponding incremental processing model for data processing according to the data characteristics of the input data, and when the incremental data arrives, call the corresponding incremental processing model for data processing cache Incremental data calculation is performed on historical processing results.

[0035] Wherein, the selection of the corresponding incremental processing model according to the data characteristics of the input data for data processing may refer ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method for incremental calculation based on MapReduce. The method includes: creating an incremental processing model for caching different historical processing results, including: caching the model of combiner results, caching the model of intermediate results, and directly reusing the results model; when the input data is obtained, select the corresponding incremental processing model for data processing according to the data characteristics of the input data, and when the incremental data arrives, call the corresponding incremental processing model for data processing and cache Incremental data calculation is performed on historical processing results. The method disclosed by the invention can save a lot of unnecessary repeated calculations by selecting a model suitable for data characteristics for calculation, thereby improving the efficiency of data processing.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a MapReduce-based incremental calculation method. Background technique [0002] With the development of the information age, more and more data are generated, and the types and scale of data are growing at an unprecedented rate. How to better manage and utilize big data has become a topic of general concern. The increase in data scale has brought great challenges to data storage, management, and data analysis. Google proposed the MapReduce model to process big data, and Microsoft also proposed a similar model, Dryad. Hadoop big data processing platform was developed on the basis of Google's open distributed file system, MapReduce model and other technical central ideas, and then the academic and business circles proposed a series of improvements around these model framework systems or proposed some new model frameworks and system. [0003] In big data processing, more and mor...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F12/0875G06F17/30
Inventor 孙广中刘惠民周英华
Owner UNIV OF SCI & TECH OF CHINA