Local computing and distributed computing based data computing method and system

Abstract

The invention relates to a local computing and distributed computing based data computing method and system. The method comprises: calling preset local computing mode and distributed computing mode to compute a same data item, marking the computing mode with relatively short computing time as a preferred computing mode of the data item, and obtaining training samples each containing data volume, the preferred computing mode and computing time information; according to the training samples, generating a training model; and estimating the data volume of a to-be-processed data item, and according to the training model and the data volume of the to-be-processed data item, determining the computing mode adapting to the to-be-processed data item and calling the computing mode to compute the to-be-processed data item. Through the data computing method and system, computing policies adapting to the scales of data items can be selected for different data items, the realization cost is low, and the computing efficiency is optimized.

Description

technical field [0001] The present invention relates to the field of computer science and technology, in particular to a data calculation method and system based on local calculation and distributed calculation. Background technique [0002] With the rapid expansion of Internet data, distributed computing is also becoming more mature. Distributed computing is a computer science that mainly studies distributed systems, such as hadoop distributed systems. A distributed system includes several computers interconnected by a network, and these computers cooperate with each other to accomplish a common goal (this common goal is called a "project"). The specific process is: divide the data items that need to be calculated into small pieces, and calculate them separately by multiple computers, and then upload the calculation results and combine them to draw data conclusions. A computer program running on a distributed system is called a distributed computing program, and distribut...

AI Technical Summary

Problems solved by technology

[0003] In order to optimize computing efficiency, on the one hand, it is to enhance the performance of local computing, or to optimize local algorithms; after optimization, for micro data projects, there is a surplus of processing capacity, which forms a waste of resources, and the implementation cost of optimization is high; on the other hand On the one hand, distributed computing is used; however, for small-scale data projects, if distributed computing is used due to the relatively large preparation time, there will be data loading delay problems

Images