Distributed data processing method and system

A distributed data and processing method technology, applied in the field of data processing, can solve the problems of occupation, long time, large data volume, etc., and achieve the effect of reducing delay, improving data transmission speed, and zero copy

Active Publication Date: 2020-06-05
XIAMEN WANGSU CO LTD
View PDF6 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since the parameter server node aggregates the data calculated by all working nodes, the amount of data is large, and the data transmission from the network card to the application layer in the parameter server in the prior art also takes a long time, which needs further optimization

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data processing method and system
  • Distributed data processing method and system
  • Distributed data processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0060] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments . Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0061] An embodiment of the present invention provides a distributed data processing method. Such as figure 1 As shown, a system architecture applicable to the embodiment of the present invention includes a parameter server 101 and a plurality of working nodes 102, wherein each working node can be a network device such as a computer or a server, and the server can be a network device such as a computer. It can be an independent device or ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention relates to the technical field of data processing, in particular to a distributed data processing method and system, which are used for improving the data transmissionspeed in a distributed system. The embodiment of the invention comprises the following steps: a network card of a main node receives to-be-processed data sent by sub-nodes, and the to-be-processed data are transmitted to a para-virtualization rear-end interface; the para-virtualization rear-end interface transmits the to-be-processed data to a para-virtualization front-end interface; the para-virtualization front-end interface transmits the to-be-processed data to an application layer; the application layer processes the to-be-processed data to obtain processed data, and the processed data aretransmitted to the para-virtualization front-end interface; the para-virtualization front-end interface transmits the processed data to the para-virtualization rear-end interface; and the network card receives the processed data transmitted by the para-virtualization rear-end interface and transmits the processed data to the sub-nodes.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a distributed data processing method and system. Background technique [0002] A distributed system is a system composed of a group of computer nodes that communicate through a network and coordinate their work to complete a common task. The emergence of distributed systems is to use cheap and ordinary machines to complete computing and storage tasks that cannot be completed by a single computer. Its purpose is to use more machines to process more data. [0003] There are two common modes of distributed training for deep learning, one is model parallelism and the other is data parallelism. For the model parallel method, due to the serious dependencies between the inner layers of the model, it is difficult to split, so the data parallel method is the mainstream method, and the PS (Parameter-Server, parameter server) architecture is more common. The PS architectu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/27G06N3/08H04L29/08
CPCG06F16/27G06N3/08H04L67/10
Inventor 陈迅
Owner XIAMEN WANGSU CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap