Spark task submitting method and device and server

A server and task technology, applied in the field of big data processing, can solve problems such as irregular use of kerberos permissions, low authentication efficiency, and prone to repetitive work

Active Publication Date: 2020-03-17
NETEASE (HANGZHOU) NETWORK CO LTD
View PDF8 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] Before submitting a spark task to a big data system, kerberos authentication is usually required to ensure data security. In related technologies, the way to perform authentication is mainly that the user adds a shell script to perform authority authentication in the spark business code. In this way, the user While implementing the spark business code, it is necessary to manually increase the authority authentication

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark task submitting method and device and server
  • Spark task submitting method and device and server
  • Spark task submitting method and device and server

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0031] In a big data system, there are many data processing tasks running. The scheduling system can manage and schedule the tasks, which greatly improves the efficiency of the big data system. Airflow is currently the most widely used task scheduling system.

[0032]In related technologies, the technical solution usually adopted for submitting spark tasks and executing permission authentication is to use the BashOperator operator of Airflow to call the spark-submit command of spark through the user's own shell script to ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a spark task submitting method, a spark task submitting device and a server. The method comprises the following steps: responding to the creation completion operation of the spark task, if the kerberos authority authentication based on the super user is passed, generating a spark-submit command containing task parameters of the spark task, and assigning an agent user name ofthe spark-sub command as a user name corresponding to a creator of a project where the spark task is located; and submitting a spark task through the spark-submit command. According to the method, kerberos permission authentication is automatically carried out while the spark task is submitted, so that the authentication efficiency is improved; the proxy user name in the spark-submit is assignedas the user name corresponding to the creator, so that the creator and the submitter of the spark task are ensured to be the same user, the user identity information is prevented from being used by other users, and the security of a big data system is ensured.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a spark task submission method, device and server. Background technique [0002] Before submitting a spark task to a big data system, kerberos authentication is usually required to ensure data security. In related technologies, the way to perform authentication is mainly that the user adds a shell script to perform authority authentication in the spark business code. In this way, the user While implementing the spark business code, it is necessary to manually increase the authority authentication logic in each spark business code, which is a heavy workload and prone to repetitive work, resulting in low authentication efficiency; moreover, this shell script-based authentication method, It is difficult to ensure the unity between the submitter of the spark task and the creator of the spark task, and it is prone to the problem of non-standard use of Kerberos permissions,...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F21/46G06F21/62
CPCG06F21/46G06F21/6227
Inventor 汪文星
Owner NETEASE (HANGZHOU) NETWORK CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap