Unlock instant, AI-driven research and patent intelligence for your innovation.

Device and method for deploying a Spark cluster on a cloud platform

A cloud platform and cluster technology, applied in the field of cloud management, can solve problems such as high requirements for ordinary developers and complex cluster deployment

Active Publication Date: 2018-12-07
ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
View PDF8 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] At present, the cluster deployment of Spark is still relatively complicated. The traditional method needs to be carried out through the Linux command line, which requires more operation and maintenance and knowledge of the Linux operating system, and has higher requirements for ordinary developers.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Device and method for deploying a Spark cluster on a cloud platform
  • Device and method for deploying a Spark cluster on a cloud platform
  • Device and method for deploying a Spark cluster on a cloud platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

[0049] The steps shown in the flowcharts of the figures may be performed in a computer system, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

[0050] In order to achieve the purpose of the embodiment of the present invention, the embodiment of the present invention provides a device 1 for deploying a Spark cluster on a cloud platform, such as figure 1 with figure 2 As shown, the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a device and a method for deploying a Spark cluster on a cloud platform. The device comprises a step management module, which is used for detecting the selection of a cluster node and a preset template of the cloud platform on a foreground interface. The cluster node may include a master node and a slave node. The default template can include: a Spark deployment template, a Spark extension template, a Spark reduction template and a Spark recycling template. The device also includes: a template management module, configured to call the corresponding cluster node and template according to the selected result; and a task management module, configured to process the Spark on the corresponding cluster node according to the retrieved template. Through the embodiment of the scheme, the deployment management threshold of the Spark cluster is lowered, the installation time is saved, and the installation configuration efficiency is improved.

Description

technical field [0001] Embodiments of the present invention relate to cloud management technology, in particular to an apparatus and method for deploying a Spark cluster on a cloud platform. Background technique [0002] Apache Spark is a fast and general computing engine designed for large-scale data processing. Spark is a Hadoop MapReduce-like general parallel framework open sourced by UCBerkeley AMP lab (AMP Lab of the University of California, Berkeley). Spark has the advantages of Hadoop MapReduce; but unlike MapReduce, the intermediate output results of Job can be saved in In the memory, there is no need to read and write HDFS (distributed file system), so Spark can be better suitable for data mining and machine learning and other MapReduce algorithms that require iteration. [0003] At present, the cluster deployment of Spark is still relatively complicated. The traditional method needs to be carried out through the Linux command line, which requires more operation a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F8/61
CPCG06F8/61
Inventor 李栋梁
Owner ZHENGZHOU YUNHAI INFORMATION TECH CO LTD