Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Machine node parallel processing method and device, computer equipment and storage medium

A parallel processing and node technology, applied in the field of artificial intelligence, can solve problems such as gradient expiration and network communication overload, and achieve the effect of alleviating gradient expiration, avoiding excessive time and storage space overhead, and reducing communication bottlenecks.

Active Publication Date: 2021-06-01
华润数字科技有限公司
View PDF4 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The purpose of the present invention is to provide a multi-machine node parallel processing method, device, computer equipment and stora

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Machine node parallel processing method and device, computer equipment and storage medium
  • Machine node parallel processing method and device, computer equipment and storage medium
  • Machine node parallel processing method and device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0029] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0030] It is to be understood that, when used in this specification and the appended claims, the terms "comprising" and "comprising" indicate the presence of the described features, integers, steps, operations, elements and / or components, but do not exclude one or The presence or addition of a number of other features, integers, steps, operations, elements, components, and / or sets thereof.

[0031] It is also to be understood that the termin...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a machine node parallel processing method and device, computer equipment and a storage medium. According to the method, after multiple machine nodes participating in neural network model training are preprocessed, the multiple machine nodes are divided into different node clusters; a machine node is selected from each node cluster to serve as a parameter synchronization temporary server of the node cluster of the machine node, and the current intra-cluster average gradient of all machine nodes of the node cluster of the machine node is calculated in each gradient exchange period; after each gradient exchange period, the parameter synchronization temporary servers of the node cluster and other node clusters exchange respective current intra-cluster average gradients so as to update model parameters of the node cluster; and finally, after a preset number of gradient exchange cycles, reselecting the parameter synchronization temporary server in each node cluster, and continuing to train the neural network model until the training is finished. The machine node parallel processing method and device, the computer equipment and the storage medium have the advantages of mitigating communication load in the network and mitigating gradient expiration.

Description

technical field [0001] The invention relates to the field of artificial intelligence, in particular to a machine node parallel processing method, device, computer equipment and storage medium. Background technique [0002] At present, the deep neural network model has become the cornerstone of the current artificial intelligence field, but most of the current models rely on the supervised learning model supported by massive data, followed by the increasing capacity of the model (in the field of natural language processing Take the model GPT3 as an example, the model parameters reach 175 billion), which brings a huge challenge to training the model; specifically, the video memory of a single graphics card can no longer accommodate the entire model for training, and distributed training needs to be performed on multiple graphics cards; The amount of sample data used for training often reaches the terabyte level, so it is necessary to distribute the data to different graphics c...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/50G06N3/04G06N3/08
CPCG06F9/5027G06F9/5072G06N3/04G06N3/08
Inventor 王伟于翠翠黄勇其
Owner 华润数字科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com