Distributed computing method and device

A technology for distributed computing and computing results, applied in the computer field, it can solve problems such as unacceptable efficiency, reduced performance, and inability to describe computing dependencies, achieving the effects of being easy to understand, reducing costs, and improving intuitiveness

Active Publication Date: 2013-11-20
BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
View PDF3 Cites 17 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the RDD-based Spark model mainly has the following problems. For the calculation of two different RDDs, Spark provides three operations: "join", "cogroup", and "crossProduct", among which "join" and "cogroup" cannot describe All calculations are dependent, and "crossProduct" will produce a lot of useless intermediate results, and its efficiency is basically unacceptable; if a calculation produces many intermediate results with the same key, only the combined calculation is performed at the end, resulting in a large number of intermediate results As a result, the storage is under a lot of pressure and the performance is significantly reduced

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed computing method and device
  • Distributed computing method and device
  • Distributed computing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0018] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention. On the contrary, the embodiments of the present invention include all changes, modifications and equivalents coming within the spirit and scope of the appended claims.

[0019] In the description of the present invention, it should be understood that the terms "first", "second" and so on are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance. In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "connected" and "connect...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed computing method and device. The method comprises the steps of generating an input vector Vector according to input data; acquiring a preset operator and the dependency of the preset operator on the Vector; carrying out multiple map computations by means of a mapping module Map till the end condition of the preset operator is met, wherein during each map computation, data are acquired from the Vector according to the dependency so that computation can be carried out according to the preset operator and corresponding data in the Vector can be updated according to computing results. The distributed computing method and device are widely used in distributed computation of various machine learning and can guarantee executing efficiency with lower cost.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a distributed computing method and device. Background technique [0002] At present, the main general-purpose computing models include the Map / Reduce model (that is, the mapping / simplification model), the model based on RDD (Resilient Distributed Datasets) transformation, and so on. [0003] For example, Hadoop (a distributed system infrastructure) uses a simple Map / Reduce model. The entire calculation process includes three parts: Map, Shuffle (shuffling), and Reduce. The upper-layer application customizes the calculation of the Map and Reduce stages. However, the Map / Reduce model mainly has the following problems. If the calculation process is very complicated, such as iterative calculation, the calculation logic will be dispersed in multiple programs (scripts), which is not intuitive, high development and maintenance costs, and long calculation time. [0004] As ano...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/44
Inventor 张健叶青
Owner BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap