Implementation method and system for sharing and using Spark cluster by multiple users in Python-Web environment

A multi-user, cluster technology, applied in the application technology implementation field of Spark cluster
CN110471777AActive Publication Date: 2019-11-19COMP NETWORK INFORMATION CENT CHINESE ACADEMY OF SCI
6 Cites 5 Cited by

Patent Information

Authority / Receiving Office
CN ยท China
Patent Type
Applications(China)
Current Assignee / Owner
COMP NETWORK INFORMATION CENT CHINESE ACADEMY OF SCI
Publication Date
2019-11-19

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
Patent Text Reader

Abstract

The invention relates to an implementation method and system for sharing and using a Spark cluster by multiple users in a Python-Web environment. Multiple processes are introduced to support multipleSpark contexts, and the limited number of Spark contexts can be used for serving and supporting task calculation requests of multiple users. Coroutines instead of threads are used for guarding sub-processes, and the coroutines are autonomously scheduled. A task calculation result or state can be returned asynchronously so as to respond to a request of a user in real time, and the requirement of Web real-time interactive use is met. The characteristic of scheduling among Spark tasks is utilized. For some tasks with long time consumption, whether the tasks are terminated or not is given to the user, and the Web user can terminate the running Spark tasks. The problem that multiple Web users share the Spark cluster in the Python development environment can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the technical field of software design and application, and relates to an application technology implementation method, in particular to an application technology implementation method and system for multi-user interactive use of Spark clusters through the Web in a Python development environment. Background technique

[0002] Apache Spark, as the most popular open source big data computing framework, is widely used in data processing and analysis applications. There are generally two ways to use it: one is to submit the user logic code packaged into a jar package through the spark-submit script; the other is to interactively process data with Spark through the spark-shell. Although these two methods are flexible enough, they are not applicable in some practical applications, such as when web users want to interact with the Spark cluster through web requests. This situation will face the problem of multiple web users sharing th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More