Efficient dynamic load balancing system and method for processing large-scale data

A large-scale data and dynamic load technology, applied in the transmission system, electrical components, etc., can solve the problems of small memory capacity, incapable of large-scale data processing, insufficient network bandwidth, etc., to achieve efficient operation, non-fixed, high-performance Effect

Inactive Publication Date: 2015-04-29
LANGCHAO ELECTRONIC INFORMATION IND CO LTD
View PDF3 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical task of the present invention is to provide a system and method for processing large-scale data with high-efficiency dynamic load balancing; realize dynamic load balancing CPU+GPU hybrid heterogeneous cluster system, make full use of the performance of the equipment, and greatly improve the efficiency of the entire system , and solve the problem that the current server computing system cannot process large-scale data due to insufficient network bandwidth and small memory capacity

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Efficient dynamic load balancing system and method for processing large-scale data
  • Efficient dynamic load balancing system and method for processing large-scale data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0036] A system for processing large-scale data with high-efficiency dynamic load balancing of the present invention is a hybrid heterogeneous cluster system of CPU and GPU, including a central control system, a computing cluster system, a storage system, and a high-speed network; the nodes in the central control system use CPU and GPU GPU hybrid heterogeneous architecture; nodes in the computing cluster system adopt CPU and GPU hybrid heterogeneous architecture or CPU architecture; storage system is divided into shared storage and local storage, shared storage nodes adopt CPU architecture, and local storage is set on the nodes of the central control system And in each node of the computing cluster system, the shared storage is divided into main storage and backup storage. The main storage and backup storage are used as redundant storage to store the same computing data. The local storage is used to store the nodes of the central control system or the Compute the data of the no...

Embodiment 2

[0040] A system for processing large-scale data with high-efficiency dynamic load balancing of the present invention is a hybrid heterogeneous cluster system of CPU and GPU, including a central control system, a computing cluster system, a storage system, and a high-speed network; the nodes in the central control system use CPU and GPU GPU hybrid heterogeneous architecture; nodes in the computing cluster system adopt CPU and GPU hybrid heterogeneous architecture or CPU architecture; storage system is divided into shared storage and local storage, shared storage nodes adopt CPU architecture, and local storage is set on the nodes of the central control system And in each node of the computing cluster system, the shared storage is divided into main storage and backup storage. The main storage and backup storage are used as redundant storage to store the same computing data. The local storage is used to store the nodes of the central control system or the Compute the data of the no...

Embodiment 3

[0044] A method for processing large-scale data with high-efficiency dynamic load balancing of the present invention uses any one of the above-mentioned systems for processing large-scale data to process large-scale data, including the following steps:

[0045] (1) The nodes in the central control system are connected to the nodes in all computing cluster systems through a high-speed network, the nodes in the central control system control the nodes in each computing cluster system, and the nodes in the central control system dynamically assign computing tasks to the computing cluster systems Node, the node in the central control system receives the return result of the node in the computing cluster system;

[0046] (2) The nodes in the computing cluster system and the nodes in the shared storage are interconnected through a high-speed network, and the nodes in the central control system and the nodes in the shared storage are interconnected through a high-speed network; Nodes...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an efficient dynamic load balancing system and method for processing large-scale data, and belongs to the technical field of processing the large-scale data. The efficient dynamic load balancing system structurally comprises a central control system, a computing cluster system, a storage system and a high-speed network. A CPU and GPU mixed heterogeneous architecture is adopted in a middle node of the central control system; a CPU and GPU mixed heterogeneous architecture is adopted in middle nodes of the computing cluster system, or a CPU architecture is adopted in nodes of the computing cluster system; the storage system is divided into a shared storage and a local storage, a CPU architecture is adopted in middle nodes of the shared storage, and the local storage is used for storing data of nodes of the central control system or the nodes of the computing cluster system; the high-speed network is used for connecting the middle node of the central control system, the middle nodes of the computing cluster system and the middle nodes of the shared storage to form the centralized system for processing the large-scale data. The problem that a current server computing system is insufficient in network bandwidth and small in storage capacity and therefore cannot process the large-scale data is solved.

Description

technical field [0001] The invention relates to the technical field of processing large-scale data, in particular to a system and method for processing large-scale data with high-efficiency dynamic load balancing. Background technique [0002] In the current social data explosion of human beings, there are more and more information data, and people have higher and higher requirements for information data processing capabilities. Not only oil exploration, weather forecast, aerospace defense, scientific research, etc. require high-performance computing, but also financial, The demand for high-performance computing in a wider range of fields such as government informatization, education, enterprises, and online games is growing rapidly. [0003] Computing speed is particularly important for high-performance computing. High-performance computing is developing toward multi-core and many-core, and heterogeneous parallelism is used to improve application computing speed. At present...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/1004H04L67/1029
Inventor 高永虎张清张广勇沈铂
Owner LANGCHAO ELECTRONIC INFORMATION IND CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap