Distributed deep learning system based on Docker container and establishment method and working method thereof

A docker container and deep learning technology, applied in the field of cloud computing virtualization, can solve the problems of resource waste, large number of hosts, and incomplete utilization, etc., and achieve the effect of simple process, cost saving, and time wasting

Active Publication Date: 2017-12-08
济南浚达信息技术有限公司
View PDF6 Cites 19 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] 1) Using the server as a stand-alone deep learning platform, the amount of CPU calculation is sufficient, but it cannot be fully utilized, resulting in waste of resources
[0006] 2) The distributed deep learning platform requires multiple hosts to build, and the CPU calculation capacity of each host is limited. If you want to build a large-scale platform, you need a large number of hosts and high costs
[0007] 3) The process of building a distributed deep learning platform is cumbersome, using the host building method, each host needs to carry out the same steps
However, this patent has the following defects: when using Dockerfile to create a mirror, it is impossible to visualize and test whether the configuration of some files in the mirror is successful

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed deep learning system based on Docker container and establishment method and working method thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0045] A distributed deep learning system based on Docker containers, such as figure 1 As shown, it includes a host machine and multiple Docker containers, Hadoop distributed platform and Spark are installed on the host machine, and the first distributed deep learning platform or the second distributed deep learning platform is also installed on the host machine; each The Hadoop distributed platform and Spark are installed on the Docker container, and the first distributed deep learning platform or the second distributed deep learning platform is also installed on each Docker container.

[0046] The server host is used as the host machine and as the hardware support of the entire platform. The first distributed deep learning platform and the second distributed deep learning platform are two currently available distributed deep learning platforms, both of which are open sourced by Yahoo, and are the current mainstream distributions. deep learning platform.

[0047] The first d...

Embodiment 2

[0051] The method for building the distributed deep learning system based on the Docker container described in Embodiment 1, the specific steps include:

[0052] (1) Prepare the host machine, which is the server host machine; install the Ubuntu 14.04 operating system; Ubuntu 14.04 is a relatively stable version of the Linux operating system that supports Docker, and you can directly install and configure the Docker environment with the command line;

[0053] (2) Create the main folder required by the Docker container in the root directory of the host machine. The main folder includes folders that can be mounted to save the training model, training data set, test data set, code and configuration files;

[0054] (3) Install the Hadoop distributed platform and Spark in the host machine; to support the CaffeOnSpark distributed deep learning platform or the TensorFlowOnSpark distributed deep learning platform; test whether the Hadoop distributed platform and Spark are installed suc...

Embodiment 3

[0064] The working method of the distributed deep learning system based on the Docker container described in embodiment 1, the specific steps include:

[0065] (1) start the Hadoop platform and Spark in the host computer, the host computer is as the master node of the whole distributed deep learning system, and start the Hadoop platform and Spark in several described Docker containers, several described Docker containers Both serve as slave nodes of the entire distributed deep learning system;

[0066] (2) Store the training model, training data set, test data set, code and configuration files required for deep learning training in the folder that can be mounted on the host computer;

[0067] (3) Start the deep learning training through the script, and the master node distributes the deep learning training tasks to each slave node for parallel training.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a distributed deep learning system based on a Docker container and an establishment method and working method thereof. The system includes a server host, a first distributed deep learning platform and a second distributed deep learning platform. By utilizing a Docker containerization technique, multiple distributed deep learning systems can be established on the server host at the same time. The improvement of the distributed deep learning system based on the Docker container mainly embodies in three aspects that (1) the whole system can be achieved in one server host without requirement of multiple hosts, so that costs are saved; (2) the container is created through a template mirror image, the processes are simple without requirement of repeated establishment, and errors and waste of time are avoided; (3) the CPU of a server can be utilized to the maximum degree, and hardware resources are no longer wasted.

Description

technical field [0001] The invention relates to a distributed deep learning system based on a Docker container and its construction method and working method, belonging to the technical field of cloud computing virtualization. Background technique [0002] In essence, cloud computing means that user terminals obtain computing resources such as storage, computing, and databases through remote connections. Virtualization technology is one of the core components of cloud computing technology, and it is a key technology to fully integrate and efficiently utilize various computing and storage resources, including server virtualization and desktop virtualization. Docker is an emerging lightweight virtualization technology. Compared with traditional VMs, it is lighter in weight and faster in startup speed. It can run hundreds or thousands of containers on a single hardware at the same time, so it is very suitable for running during peak business hours. Launch a large number of con...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/445H04L29/08
CPCG06F8/63H04L67/10
Inventor 张舒吴大雷张秀真
Owner 济南浚达信息技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap