Method for generating and executing dynamic spark task

Abstract

The invention discloses a method for generating and executing a dynamic spark task. The method comprises the following steps that a user initiates a task creation request to a service verification module; the service verification module distributes the task creation request to a dynamic task generation service; the dynamic task generation service obtains a task template and fills parameters to generate a Python script, and stores the Python script in hdfs; the hdfs returns the generated script file name and the hdfs path to the dynamic task generation service; the dynamic task generation service obtains the Python script in the hdfs and submits the Python script to a cluster for execution; the cluster obtains a task execution state and returns the task execution state to the dynamic task generation service; and the dynamic task generation service returns a task execution result to the user. According to the method, the spark execution script can be automatically generated and submitted to the cluster for execution as long as the data source and the data cleaning condition are provided, web interactive development and dynamic generation of tasks are realized, and the availability is high.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method for generating and executing a dynamic spark task. Background technique [0002] In the process of big data development, developers often write spark tasks in scala, then package them into jars and submit them to the cluster for operation. This big data development method is common and effective, but every time there is a new spark task, the developer needs to rewrite the scala program and package it and submit it to the cluster. This process is cumbersome, and the generation of the task still requires the developer to write code , It is necessary to understand big data related technologies, and users are also required to maintain big data related components. It is undoubtedly a burden for teams who do not care about how big data is executed and only expect to obtain valuable data. Therefore, it is necessary to provide a method for generating and executing dynam...

AI Technical Summary

Problems solved by technology

This big data development method is common and effective, but every time there is a new spark task, the developer needs to rewrite the scala program and package it and submit it to the cluster. This process is cumbersome, and the generation of the task still requires the developer to write code , needs to understand big data-related technologies, and also requires users to maintain big data-related components. It is undoubtedly a burden for a team that does not care about how big data is executed and only expects to obtain valuable data.

Images