Data processing method, device and system

A data processing and data technology, applied in the field of distributed computing, can solve problems such as network congestion, large data transmission volume, and affecting computing efficiency, and achieve the effects of path length balance, high computing efficiency, and improved execution efficiency

Active Publication Date: 2019-09-13
HUAWEI TECH CO LTD
View PDF13 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] This application provides a data processing method, device and system, which can solve the problem of data transmission volume in the network when the data center network in the related art realizes distributed computing. Larger, there may be network congestion, affecting computing efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device and system
  • Data processing method, device and system
  • Data processing method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0066] Figure 1A It is an architectural diagram of the data center network involved in the data processing method provided by the embodiment of the present invention, as shown in Figure 1A As shown, the data center network may include a controller 01 , multiple computing nodes 02 and at least one switching device 03 for connecting the multiple computing nodes 02 . Wherein, the controller 01 and each computing node 02 can be deployed in a server, and the switching device 03 can be a switch with data forwarding and data processing functions. refer to Figure 1A It can be seen that the controller 01 establishes a communication connection with each switching device 03 , and any two computing nodes 02 can establish a communication connection through the switching device 03 .

[0067] In the embodiment of the present invention, the plurality of computing nodes 02 can be used to implement distributed computing tasks such as distributed machine learning, for example, the plurality of...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data processing method, device and system, and relates to the field of distributed computing. The method comprises: a controller receiving a processing request which is sent by a designated node and carries identifiers of a plurality of computing nodes used for executing a designated computing task, and then a target switching device can be determined from switching devices used for being connected with the plurality of computing nodes; respectively sending routing information used for indicating a data forwarding path between the plurality of computing nodes and the target switching equipment to the target switching equipment and the specified node; wherein the target switching device is used for carrying out merging processing on the data reported by the plurality of computing nodes according to the routing information and then sending the data to each computing node, the specified node is used for sending the routing information to each computing node, and each computing node can report the data to the target switching device according to the routing information. According to the method provided by the invention, the probability of network congestion canbe reduced, and the calculation task execution efficiency is improved.

Description

technical field [0001] The present application relates to the field of distributed computing, in particular to a data processing method, device and system. Background technique [0002] Distributed machine learning generally uses data parallelism for model training. When using this data parallel method for model training, each computing node (also called worker) stores an algorithm model, and each node can obtain part of the sample data separately, and can perform training on the obtained sample data Get the model parameters. Each calculation node needs to send the calculated model parameters to the parameter server (parameter server, ps), which is used to aggregate and update the model parameters reported by each calculation node, and then send the updated model parameters to each calculation node. node. [0003] In related technologies, a high-performance computing (High Performance Computing, HPC) data center network is usually used to implement distributed machine lea...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/733H04L12/741H04L12/801H04L12/891H04L45/122H04L45/02H04L45/74H04L47/41
CPCH04L45/20H04L45/74H04L47/41H04L45/122H04L45/64H04L45/42H04L45/306H04L45/025H04L67/10H04L67/63G06N3/08H04L45/127H04L47/12
Inventor 黄伊夏寅贲刘孟竹
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap