Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A Reliable Data Analysis Method Based on RDMA and Message Passing

A technology of data analysis and message transmission, which is applied in the direction of data error detection, electrical digital data processing, and response error generation to achieve the effects of improving system performance, simplifying program writing, and having fault tolerance.

Active Publication Date: 2020-12-04
YANGTZE DELTA REGION INST OF TSINGHUA UNIV ZHEJIANG
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But there is currently no streamlined and optimized message communication library for distributed machine learning program development

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Reliable Data Analysis Method Based on RDMA and Message Passing
  • A Reliable Data Analysis Method Based on RDMA and Message Passing
  • A Reliable Data Analysis Method Based on RDMA and Message Passing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] The implementation method of the present invention will be described in detail below, and the implementation described with reference to the accompanying drawings is exemplary, and is only used to explain the present invention, and should not be construed as limiting the present invention. A reliable data analysis method based on RDMA and message delivery according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0024] figure 1 It is the functions and execution flow of each part implemented according to the present invention. figure 2 It is actually the flow of function calling required by the method of the present invention. Such as figure 1 shown, combined with figure 2 , the method includes the following steps:

[0025]Step S1: Provide Allreduce and Broadcast functions, and perform data reduction calculation of cluster nodes according to Allreduce function, and perform data broadcast operation of ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a reliable data analysis method based on RDMA and message passing. The method comprises the following steps that an Allreduce function and a Broadcast function are provided, data reduction calculation of cluster nodes is conducted according to the Allreduce function, and a data broadcast operation of the cluster nodes is conducted according to the Broadcast function; a fault-tolerant mechanism based on an application-level checkpoint technology is generated to conduct backup and restoration of data; backup and restoration of local data is conducted through an RDMA technology, wherein during data backup, the local data is directly written in corresponding addresses of multiple remote machines by using unilateral writing characteristics of the RDMA technology, and during data restoration, data is directly written in from the remote machines to restore local duplicate spaces by using the unilateral writing characteristics of the RDMA. The reliable data analysis method can simplify programming and improve the calculation efficiency, meet a part of calculation requirements of distributed machine learning programs and has fault tolerance.

Description

technical field [0001] The invention relates to the technical field of parallel computing, in particular to a reliable data analysis method based on RDMA and message delivery. Background technique [0002] With the massive growth of information data, more and more enterprises need to convert messy and disorderly big data into valuable information. At present, the original big data is basically unstructured data, and it is difficult to directly infer the law from it. Certain data analysis methods process data to obtain valuable conclusions. [0003] A machine learning algorithm is an algorithm that obtains the law from the original data through continuous iterative learning operations, and uses the law to predict the conclusion of the unknown data. It has important applications in the fields of data mining, computer vision, and natural language processing. . Large-scale and its learning are of great significance to companies, governments and institutions, which also means t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F11/14
CPCG06F11/1441G06F11/1458G06F11/1464G06F11/1469
Inventor 陈康武永卫王彬
Owner YANGTZE DELTA REGION INST OF TSINGHUA UNIV ZHEJIANG
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com