Big data task scheduling method

A task scheduling and big data technology, applied in digital data processing, program startup/switching, resource allocation, etc., can solve problems such as poor utilization of computing resources, and achieve the effect of solving excessive resource preemption and full utilization

Pending Publication Date: 2021-01-22
SHENZHEN GRADUATE SCHOOL TSINGHUA UNIV
View PDF5 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide a big data task scheduling method to solve the problem of poor utilization of computing resources in the prior art when performing big data task analysis, and realize the use of the least number of machines to complete the most big data analysis business

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data task scheduling method
  • Big data task scheduling method
  • Big data task scheduling method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0029] like figure 1 As shown, a big data task scheduling method includes the following steps:

[0030] S1. Divide multiple big data analysis tasks into multiple priorities according to their importance, each big data analysis task has its own priority, divide the same priority big data analysis tasks into the same group, and get multiple group task group.

[0031] In step S1, the priority of each big data analysis task can be divided according to its business importance, and the higher the importance, the higher the priority.

[0032] In step S1, the priority of each big data analysis task can also be divided according to the importance of business guidance according to its analysis conclusion, and the higher the importance, the higher the priority.

[0033] Among them, each step is explained as follows:

[0034] S2. Determine the complexity of eac...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a big data task scheduling method, which comprises the following steps of S1, dividing a plurality of big data analysis tasks into a plurality of priorities, dividing the big data analysis tasks with the same priority into the same group, and determining the complexity of each big data analysis task in each task group; and S2, constructing a task scheduling subroutine basedon a cyclic scheduling learning algorithm neural network in the Hadoop computing cluster, and allocating computing resources of the Hadoop computing cluster to each big data analysis task by the taskscheduling subroutine according to priorities and complexity. According to the method, the computing cluster can reach the optimal running state during big data analysis, the problem of excessive resource preemption of the computing task is solved, and meanwhile, the computing resources of the Hadoop cluster are recycled in time, so that the computing resources are fully utilized.

Description

technical field [0001] The invention relates to the field of big data intelligent processing methods, in particular to a big data task scheduling method. Background technique [0002] When the world is striding into the 5G era, data is increasingly becoming a gold mine for enterprises. To extract the desired gold from these data gold mines, it is necessary to use big data analysis technology and the powerful computing power of server clusters to Get a variety of data reports, so that through these reports, you can intuitively have a clearer understanding and understanding of related businesses. With the increase of data volume, from the beginning of GB to TB, even to PB-level data, a very large big data cluster is needed to meet the data analysis requirements, and the analysis requirements also range from a few to dozens to several Hundreds. [0003] At present, in the field of big data analysis, it is necessary to collect non-sensitive behavioral data of users as permitte...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/50
CPCG06F9/4881G06F9/5027G06F9/5066
Inventor 胡亚军邵若梅孙树清
Owner SHENZHEN GRADUATE SCHOOL TSINGHUA UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap