Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Task priority control implementation method and device for Spark JDBC

A technology of task priority and implementation method, applied in the field of task priority control for SparkJDBC, can solve problems such as affecting business applications, unable to schedule resources, unable to retrieve SQL priority control, etc., to achieve the effect of improving usability and large application prospects

Inactive Publication Date: 2019-06-07
NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF6 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, with the continuous increase of data volume and the continuous development of big data technology, SparkJdbc's native architecture cannot flexibly schedule resources, and cannot perform priority control on retrieval SQL. This problem directly affects business applications.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Task priority control implementation method and device for Spark JDBC
  • Task priority control implementation method and device for Spark JDBC
  • Task priority control implementation method and device for Spark JDBC

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] The invention provides a method for realizing Spark JDBC-oriented task priority control. Including the method of establishing multiple task priority queues in SparkJDBC; the method of mapping the query SQL submitted by the user to the task priority queue in Spark JDBC to wait for execution; setting the execution limit for each task priority queue in Spark JDBC, The method of refusing to retrieve SQL beyond the limit; the method of scheduling hardware resources according to the preset priority and weight between task priority queues; within a single task priority queue, use the "first in first out" strategy or "fair" strategy A method for scheduling hardware resources. Using the embodiment of the present invention can satisfy the requirement of flexibly controlling the use of hardware resources through the JDBC interface in actual business use; and meet the requirements for the execution timing of urgent tasks, general tasks and low-priority tasks in business use.

[00...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a task priority control implementation method and device for Spark JDBC. The method comprises the steps of describing an XML file by a Spark Jdbc service according to a pre-written priority queue to establish a plurality of task priority queues when started; receiving a specified priority queue command issued by a user through a Jdbc interface, and completing priority setting of a Jdbc session level; receiving a retrieval SQL submitted by a user, generating a Spark Task set after the SQL statement is subjected to a plurality of analysis planning processes, and adding the Spark Task set into a target priority queue of a corresponding name; and scheduling and allocating hardware resources through a resource scheduler according to the resource allocation strategy between the priority queues and the resource allocation strategy in the queues, and allocating Spark Tasks to Task actuators on the computing nodes for execution.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a method and device for implementing Spark JDBC-oriented task priority control. Background technique [0002] With the continuous development of computer technology and the continuous improvement of informatization, the amount of data has grown rapidly, and the storage and application of massive data has also flourished. In massive data retrieval applications, Apache Foundation's distributed retrieval framework SparkJdbc provides a HiveQL interface with Hive, which has high efficiency and availability and is widely used in this field. [0003] After the user submits a SQL retrieval request to SparkJdbc, the SQL statement is parsed to generate an execution plan, and then a SparkRDD is generated. The Spark RDD is converted into a DAG to generate a Spark Stage, and finally the Stage generates a Spark Task collection. Spark Task is a task structure generated in Spark that can perf...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/48G06F9/50G06F16/242G06F16/25
Inventor 刘欣然张鸿惠榛吕雁飞马秉楠李斌斌王振宇黄航王树鹏
Owner NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com