A hive task execution engine selection method and system
An execution engine and task execution technology, applied in the computer field, can solve problems such as increased labor burden, task delay, error, etc., and achieve the effect of reducing failure risk, avoiding task delay, and reducing human resources
Active Publication Date: 2022-03-25
BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
View PDF5 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0012] 1. A large number of tasks in the scheduling system are manually configured, which undoubtedly increases the labor burden. In reality, everyone has development tasks, and there is not so much manpower to configure each task, and after configuration, it is necessary to observe whether the task is executed. shortened time
[0013] 2. Users need to record which tasks are configured and which tasks are not configured, and need to set configuration plans and other work
Moreover, if the parameter configuration is wrong in the configuration, it will cause a new risk of failure
[0014] 3. Since the new configuration may cause the task to run incorrectly, if the new configuration is incorrect, the error will still occur after automatic retry
Manual processing must be performed after receiving the alarm information, which may cause a large number of task delays
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0057] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.
[0058] figure 1 A schematic flow chart of the HIVE task execution engine selection method provided for some embodiments of the present application, such as figure 1 shown, including the following steps:
[0059] Step S11, calculating the critical path in the task topology map;
[0060] Preferably, the critical path is calculated according to task dependenc...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present application provides a method and system for selecting an HIVE task execution engine. The method includes calculating a critical path in a task topology graph; calculating the delayable time of each task on the critical path, and selecting available new The task executed by the selected execution engine; the new execution engine is tested for the selected task, and the execution engine of the selected task is determined according to the test result; the above steps are repeated until the execution engine of each task is determined. The human resources required for task configuration of the scheduling system are reduced, the failure risk of parameter configuration errors is reduced, and the task delay caused by configuration errors is avoided.
Description
【Technical field】 [0001] The present application relates to the field of computer technology, in particular to a HIVE task execution engine selection method and system. 【Background technique】 [0002] Hive is an open source software that uses Structured Query Language (SQL) commonly used by data analysts to process data in distributed storage systems. [0003] There are three types of execution engines in Hive. The longest, most widely used, and most stable execution engine is MapReduce. MapReduce is the distributed processing framework that comes with Hadoop, an open source big data processing platform. When Hive uses MapReduce to execute SQL tasks, zero or more MapReduce tasks are required to run the final result. If the amount of data calculated by SQL is relatively small and does not exceed a certain value, such as 10M (configurable by parameters), then Hive can load these data into the memory where Hive is executed for calculation. If the amount of data is relatively...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2455G06F16/2458
Inventor 侯志贞吴庆鑫
Owner BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD