Spark semantics based data reuse method and system thereof

A data and semantic technology, applied in the field of data reuse method and system based on Spark semantics, can solve the problems of no reuse cache data migration function, inability to perform automatic cache operation, low data reuse rate, etc., to reduce repeated data calculation. , Increase the memory capacity of the cluster, and accelerate the effect of computing

Active Publication Date: 2017-03-08
POWERLEADER TELECOM TECH
View PDF5 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the purpose of the present invention is to provide a data reuse method and system based on Spark semantics, aiming to solve the problem of inability to perform heuristic automatic cache operati

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark semantics based data reuse method and system thereof
  • Spark semantics based data reuse method and system thereof
  • Spark semantics based data reuse method and system thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0021] In the technical solution provided by the present invention, in order to improve the efficiency of data reuse in Spark and accelerate calculation, it is necessary to design a heuristic data caching and migration mechanism in Spark that can adapt to mixed memory media. The reason is that in the application of this iterative computing framework, the more repeated operations are required, the larger the amount of data to be read, and the greater the benefit of computing. Machine learning algorithms and interactive queries are typical applications that require Cache operation on reused data; non...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention provides a Spark semantics based data reuse method. The method comprises a semantic information collection step of collecting semantic information during the operation of a Spark application; a semantic maintenance step of maintaining the semantic information collected from the semantic information collection step; a data active caching step of caching data that is not explicitly cached by a user program according to the semantic information and a preset threshold model; and cached data migrating step of carrying out migration on the cached data between a dynamic random access memory and a fixed memory according to the semantic information and the preset threshold model. The present invention further provides a Spark semantics based data reuse system. The technical scheme provided by the present invention can reduce the repetitive data calculation, improve the calculation efficiency, and effectively avoid the dependence on the experience of developers.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a data reuse method and system based on Spark semantics. Background technique [0002] Spark is currently an efficient and widely used big data computing framework in the industry. It is especially suitable for applications with multiple iterative calculations, such as machine learning, graph processing, data mining, and interactive query. Data is cached in memory. Spark is more versatile and flexible than Hadoop, providing users with a variety of operators, and users can name, materialize, and control the storage and partitioning of intermediate results, which provides convenience for users who develop upper-layer applications. [0003] In addition, the current performance bottleneck of Spark has shifted to CPU and memory, and large-capacity memory will effectively improve Spark's memory computing efficiency. Non-volatile memory has the characteristics of large capacity, fast re...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/38G06F9/30G06F3/06
CPCG06F3/0655G06F9/30098G06F9/3867
Inventor 陆克中毛一帆黄泽成王明俭毛睿廖好
Owner POWERLEADER TELECOM TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap