Tensorflow-based multi-task elastic scheduling method and tensorflow-based multi-task elastic scheduling system

A scheduling method and multi-task technology, applied in the computer field, can solve problems such as program errors, increased trouble, and low usability

Pending Publication Date: 2021-03-09
WUHAN INSTITUTE OF TECHNOLOGY
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

It introduces the idea of ​​multi-tenancy and flexible scheduling, but the way it expands from the inside of the cluster will change the composition of the cluster, so that it is difficult for users to make in-depth adjustments, and at the same time, some programmatic errors may occur, adding unnecessary trouble , less available

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Tensorflow-based multi-task elastic scheduling method and tensorflow-based multi-task elastic scheduling system
  • Tensorflow-based multi-task elastic scheduling method and tensorflow-based multi-task elastic scheduling system
  • Tensorflow-based multi-task elastic scheduling method and tensorflow-based multi-task elastic scheduling system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0036] A multi-task flexible scheduling method and system based on tensorflow according to the embodiment of the present invention, such as figure 1 shown, including the following steps:

[0037] S1: Use the task management system to preprocess the tasks entering the cluster, allowing multiple tasks to be performed simultaneously in the cluster;

[0038] Among them, a small amount of training code program needs to be modified first, and the configuration file of the task management system is referenced. Users can modify configuration files on the task management system to set up the tensorflow tr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a tensorflow-based multi-task elastic scheduling method and system, and the method comprises the following steps: carrying out the preprocessing of tasks entering a cluster through a task management system, and allowing a plurality of tasks in the cluster to be carried out at the same time; obtaining the number of all GPU resources in the cluster from a configuration file of the task management system, dividing a data set for the training task by utilizing the number of available GPUs, and dividing each part of data set to a specified GPU; elastically allocating video memory resources to the tasks on each GPU, and elastically expanding idle GPUs to improve the training speed; completing gradient descent on each part of data set to obtain the current gradient of eachpart; accumulating the gradients by using the communication among the clusters to obtain a current total gradient; broadcasting the total gradient to each GPU, and then performing next gradient descent. According to the method, multiple tasks can be elastically scheduled to enter the cluster, and distributed computing is efficiently completed by utilizing existing resources of the cluster.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a multi-task flexible scheduling method and system based on tensorflow. Background technique [0002] As an important branch of the field of machine learning, deep learning has received high attention from industry and academia in recent years, and has achieved remarkable development, and has been widely used in machine vision, speech recognition and other fields. However, massive training data and ultra-large-scale models have brought increasingly severe challenges to deep learning. Distributed deep learning has emerged as the times require and has gradually become an effective means to deal with this challenge. [0003] Google has developed the MapReduce system, which has achieved certain results in the distributed training of processing large-scale data. This framework realizes the ability to efficiently process large-scale data, but its ability in deep learning ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50G06T1/20G06N20/00
CPCG06F9/4843G06F9/5027G06T1/20G06N20/00
Inventor 李迅周覃张彦铎尹健南王重九崔恒
Owner WUHAN INSTITUTE OF TECHNOLOGY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products