Method for generating and executing dynamic spark task

A dynamic task and task execution technology, applied in the field of data processing, can solve cumbersome problems and achieve the effects of reduced maintenance costs, high availability, and easy management and maintenance

Active Publication Date: 2021-09-28
深圳市信润富联数字科技有限公司
View PDF7 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This big data development method is common and effective, but every time there is a new spark task, the developer needs to rewrite the scala program and package it and submit it to the cluster. This process is cumbersome, and the generation of the task still requires the developer to write code , needs to understand big data-related technologies, and also requires users to maintain big data-related components. It is undoubtedly a burden for a team that does not care about how big data is executed and only expects to obtain valuable data.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for generating and executing dynamic spark task
  • Method for generating and executing dynamic spark task

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0028] refer to figure 1 and figure 2 As shown, the present invention provides a method for generating and executing a dynamic spark task, comprising the following steps:

[0029] S1, the user initiates a spark task creation request to the service verification module, thereby passing the target data source and data processing rule parameters to the service verification module; that is, the spark task creation request carries the target data required to generate the spark task Source and data processing rule parameters, when the user service verification module sends a spark task creation request to the service verification module, the target data source and data processing rule parameters are synchronously passed to the service verification module;

[0030] Wherein, the service verification module is used to implement authority verificatio...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for generating and executing a dynamic spark task. The method comprises the following steps that a user initiates a task creation request to a service verification module; the service verification module distributes the task creation request to a dynamic task generation service; the dynamic task generation service obtains a task template and fills parameters to generate a Python script, and stores the Python script in hdfs; the hdfs returns the generated script file name and the hdfs path to the dynamic task generation service; the dynamic task generation service obtains the Python script in the hdfs and submits the Python script to a cluster for execution; the cluster obtains a task execution state and returns the task execution state to the dynamic task generation service; and the dynamic task generation service returns a task execution result to the user. According to the method, the spark execution script can be automatically generated and submitted to the cluster for execution as long as the data source and the data cleaning condition are provided, web interactive development and dynamic generation of tasks are realized, and the availability is high.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method for generating and executing a dynamic spark task. Background technique [0002] In the process of big data development, developers often write spark tasks in scala, then package them into jars and submit them to the cluster for operation. This big data development method is common and effective, but every time there is a new spark task, the developer needs to rewrite the scala program and package it and submit it to the cluster. This process is cumbersome, and the generation of the task still requires the developer to write code , It is necessary to understand big data related technologies, and users are also required to maintain big data related components. It is undoubtedly a burden for teams who do not care about how big data is executed and only expect to obtain valuable data. Therefore, it is necessary to provide a method for generating and executing dynam...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/448G06F8/30
CPCG06F9/4488G06F8/315
Inventor 覃江威杜冬冬罗启明熊皓杨志宇吴育校成建洪陈功陈军冯建设
Owner 深圳市信润富联数字科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap